GitOps Implementation Guide: From First Commit to Production

What if your entire production environment could be rebuilt from a single Git commit? That is the promise of GitOps, and it is why so many platform teams have rebuilt their delivery pipelines around it. The idea sounds simple: Git holds the desired state, an agent inside the cluster makes reality match it. The practice is messier. Teams hit the same walls in the same order: secrets, environment promotion, drift, and the uncomfortable fact that whoever controls the repository controls the cluster. This guide walks through the whole journey, from the first principle to the hardening checklist, with the practical details that separate a working setup from a demo.

What GitOps Actually Means in Practice

GitOps is an operational framework that treats Git as the single source of truth for declarative infrastructure and applications. Every change starts as a commit. An automated agent running inside the cluster watches the repository and reconciles the live state toward what the repository declares. The term gained traction around 2017, and the definition has stayed remarkably stable since then.

The OpenGitOps project, a community effort under the Cloud Native Computing Foundation, distilled the model into four principles that are worth memorizing because they settle most design arguments later:

• Declarative. The system describes what it wants, not how to get there.

• Versioned and immutable. Every desired state has a recorded history in Git.

• Pulled automatically. An agent fetches the state rather than receiving a push.

• Continuously reconciled. The agent keeps checking and correcting.

That third principle is where most misunderstandings live. In a push model, a CI system outside the cluster applies changes using credentials it holds. In a pull model, the agent lives inside the cluster and reaches out to Git, so cluster credentials never leave the cluster. The pull model is what the principles actually require, and it changes your security posture in a way that matters more than most teams expect.

If you want the primary definition rather than a summary, the OpenGitOps principles documentation lays out all four with the reasoning behind each.

Choosing Your Controller: Argo CD or Flux

Two controllers dominate the Kubernetes GitOps space, and both are graduated projects in the Cloud Native Computing Foundation. The choice between them is less about features than about philosophy.

Neither is a mistake. Teams that need non-engineers to see deployment status tend to gravitate toward Argo CD. Teams that treat everything as code and dislike dashboards tend to prefer Flux. What matters far more than the choice is consistency: pick one, learn its reconciliation model properly, and avoid running both against the same resources.

Whichever you pick, read the official Argo CD documentation or the Flux documentation end to end before you design your repository layout. Both projects document their reconciliation behavior in detail, and the details are where surprises hide. (Kubernetes Documentation: Overview)

Designing the Repository Layout

Repository structure is the decision teams most often regret. There are three common patterns, and each has a clear use case.

Repository per environment

Separate repositories for staging and production give you strong access control. A junior engineer can merge to staging without touching production. The cost is duplication: shared configuration has to be copied or referenced across repos, and keeping them in sync becomes its own chore.

Directory per environment

One repository with a directory tree for each environment is the simplest starting point. Promotion happens by copying a manifest from one directory to the next, usually through a pull request. It works well for a handful of services and a couple of environments. It gets unwieldy past that.

Environment branches

Branches per environment look tidy but age badly. Merging between long-lived branches creates conflicts, and the branch history stops reflecting what actually runs. Most experienced teams move away from this pattern after their first painful merge.

For most organizations, a hybrid works best: an application repository holding service code and base manifests, plus a separate environment repository holding overlays and cluster-specific values. Promotion becomes a pull request against the environment repository, which gives you a review trail for free.

For multi-cluster setups, templated fan-out is the mechanism to learn. ApplicationSets in Argo CD became generally available in version 2.3, and Flux offers equivalent composition through Kustomization and HelmRelease resources. The ApplicationSet documentation explains how to generate one application definition per cluster from a single template.

The Reconciliation Loop and Drift

The reconciliation loop is the heartbeat of any GitOps system. The controller continuously compares the desired state in Git against the live state in the cluster and corrects differences. Argo CD calls this a sync, and it deliberately separates two statuses: sync status tells you whether the live state matches Git, and health status tells you whether the workload is actually working. A resource can be perfectly in sync and completely unhealthy at the same time, and conflating the two is a common source of confusion during incidents.

Drift is the enemy the loop exists to fight. Someone edits a deployment by hand during an outage. A webhook mutates a resource. A namespace gets a default injected that nobody declared. The controller catches all of it and reverts, which is exactly what you want, except when it is not. Practical advice: enable automatic sync for most workloads, but leave manual sync available for production databases and anything with a slow rollback path. And when someone needs to make an emergency change, teach them to commit it rather than kubectl it, because the loop will undo a hotfix faster than they can document it.

Handling Secrets Without Wrecking the Model

Plain Git is not a safe place for secrets. This is the model’s best-known gap, and every implementation has to solve it. The common approaches each trade off differently:

• Sealed Secrets encrypts secrets so only the cluster controller can decrypt them. Simple, but rotation is manual.

• SOPS with age or GPG encrypts values in place, so the file stays readable in structure. Works well with existing workflows.

• External Secrets Operator keeps secrets in an external store and syncs them into the cluster as native objects. Best for teams already running a vault.

• HashiCorp Vault with an injection mechanism gives the strongest control but adds operational weight.

The External Secrets Operator documentation explains the sync model and is a good starting point if you already have a secrets manager. The practical mistake here is treating secret management as an afterthought and committing a base64-encoded value “just for now.” That commit lives in history forever, and rotating the credential is the only real fix.

Progressive Delivery Alongside GitOps

GitOps handles what should run. It does not, by itself, decide whether a new version is safe. That is where progressive delivery comes in: canary releases and blue-green deployments that shift traffic gradually and roll back automatically when metrics degrade.

Argo Rollouts and Flagger are the two common tools, and both integrate cleanly with a GitOps controller. The pattern is straightforward: the controller syncs the new manifest, the rollout controller manages the traffic shift, and the GitOps controller reports health based on the rollout’s state. The Argo Rollouts documentation covers the analysis templates that automate the go or no-go decision. Without that automation, progressive delivery is just a slower deployment.

Security: The Repository Is Now the Cluster

Here is the uncomfortable truth about pull-based GitOps: the controller has broad rights inside the cluster, often cluster-admin. That makes the Git repository a high-value target. Anyone who can merge to the production branch can, in effect, run workloads with those rights.

Hardening follows from that. Branch protection with required reviews is not optional. Signed commits give you a verifiable trail. Least-privilege role-based access control for the controller limits the blast radius if it is compromised, and restricting which namespaces it can write to does the same. The Flux security documentation walks through the threat model for a pull-based controller and is worth reading even if you run Argo CD, because the risks are structural rather than tool-specific.

One more habit that pays off: audit what the controller can reach, not only what it currently manages. A controller with permission to write to every namespace will eventually write to one you did not plan for.

A Practical Rollout Sequence

Teams that succeed with GitOps tend to follow a similar order. Skipping steps is what produces the half-migrated setups that nobody trusts.

• Start with one non-critical service. Pick something with a fast rollback and low blast radius.

• Define the repository layout before the second service arrives. Retrofitting structure is painful.

• Install the controller and connect it read-only first. Watch what it would change before you let it change anything.

• Enable automatic sync for stateless workloads. Keep manual sync for stateful ones.

• Solve secrets before onboarding anything that needs them. Not after.

• Add progressive delivery for customer-facing services. Start with a simple canary and one metric.

• Harden access. Branch protection, signed commits, scoped permissions.

• Migrate the rest of the estate service by service. Track progress, because a stalled migration leaves two competing sources of truth.

Common mistakes cluster around steps three and eight. Teams enable automatic sync on day one and get surprised by a controller reverting a manual change during an incident. Or they migrate most services and leave the last few managed by hand, which means the repository no longer tells the truth about production, and the whole model quietly loses credibility.

Where GitOps Fits in a Wider Platform

GitOps is one layer of a platform, not the whole thing. It pairs naturally with infrastructure-as-code for the cluster itself, policy engines that validate manifests before they sync, and observability that ties deployments to outcomes. For teams thinking about how delivery fits into broader infrastructure work, the cloud coverage on 9481.org is a useful companion read, and the blog section covers adjacent operational topics. It also helps to follow industry news on platform tooling as the ecosystem around these controllers moves quickly.

One honest caveat: there is no formal ISO or IETF standard for GitOps. The OpenGitOps principles are the closest thing to a normative definition, and they are a community specification rather than a ratified standard. That is fine for adoption, but it means vendor claims about “GitOps compliance” deserve scrutiny.

Conclusion

GitOps works because it replaces a pile of imperative scripts with one auditable truth. The implementation, though, is a series of decisions that compound: which controller, which repository layout, how secrets are handled, how much the controller is allowed to touch. Get the four principles right and the rest becomes engineering rather than guesswork. Start small, structure early, solve secrets before they bite, and treat the repository with the same seriousness you treat production. Because in a pull-based system, it is production.

Frequently Asked Questions

What is the difference between GitOps and CI/CD?

CI/CD describes the pipeline that builds and tests code. GitOps describes how the resulting desired state reaches the cluster. They complement each other: CI produces artifacts and updates manifests, while the GitOps controller handles deployment and ongoing reconciliation. A GitOps setup usually still has CI, just with a narrower job. (GitOps)

Do I need Kubernetes to use GitOps?

No, but the tooling is overwhelmingly Kubernetes-focused. The principles apply to any declarative system, and some teams apply them to network devices or virtual machines. In practice, the mature controllers and community patterns are built around Kubernetes, so that is where most implementations start.

Is GitOps secure if the repository holds production configuration?

It can be, provided you treat the repository as a privileged asset. That means branch protection, required reviews, signed commits, and scoped permissions for the controller. Secrets should never sit in plain text; use an encryption or external-store approach instead. The repository becomes the attack surface, so secure it like one.

How do I roll back a bad deployment with GitOps?

Revert the commit. The controller detects the change and reconciles the cluster back to the previous desired state. This is one of the model’s strongest features, but it depends on the previous state being healthy and on the rollback path being fast enough. For stateful workloads, test the revert before you need it.

Can GitOps and manual changes coexist?

They can, but not comfortably. The reconciliation loop will revert manual changes, which is the point. The workable compromise is manual sync for selected sensitive workloads, so a human approves the application of changes. Anyone making an emergency edit should commit it immediately afterward to keep history honest.

Which controller should a small team start with?

Argo CD is usually the easier first step because its dashboard makes sync and health status visible without extra tooling. Flux suits teams that prefer a CLI-first, composable approach. Either works; the bigger factor is committing to one and learning its reconciliation behavior properly rather than switching later.