What happens when the team that clicks “create cluster” at 2 a.m. is the only one who knows which GPU settings actually work? That question is central to AI infrastructure as code. Machine learning platforms fail in ways ordinary web apps do not: a missing device plugin, an unapproved instance type, or an unsigned model artifact can stall a training run for hours. Writing the whole stack down as versioned files makes that informal knowledge reviewable, testable, and rebuildable.
This article explains how IaC practices apply to AI systems, where the usual cloud patterns break down, and what actually works in practice.
What AI infrastructure as code actually means
Infrastructure as code (IaC) means managing compute, network and storage through machine-readable definition files instead of manual configuration. The practice is widely traced to a 2006 talk by Werner Vogels, then Amazon’s CTO, though the phrase spread far beyond that origin.
Applied to AI, IaC extends the same idea to the parts that make machine learning different: GPU pools, high-bandwidth networking, model servers, vector stores, and the pipelines that move data and weights around. The goal is simple to state. Anyone with the right permissions should be able to recreate the training environment from a repository, and get the same result.
The tooling is familiar. Terraform, first released in 2014, uses providers to talk to cloud APIs, and its registry hosts thousands of them, including ones for GPU clouds and Kubernetes. For a broader look at how the chip layer shapes these decisions, see AI chips and computing infrastructure.
Why AI workloads strain ordinary IaC patterns
Standard cloud infrastructure is mostly fungible. One virtual machine is much like another. AI infrastructure is not. Three differences matter most.
Scarcity. GPU capacity is limited and expensive. A plan that creates twenty A100 nodes may succeed technically and fail commercially. IaC for AI has to encode quotas, approved instance families, and fallback regions in addition to resource definitions.
Statefulness. Training jobs checkpoint to storage. Model weights are large binary artifacts with their own lifecycle. A destroyed cluster is not always recoverable from a fresh apply, because the interesting state lives in a checkpoint bucket.
Hardware coupling. Drivers, CUDA versions, and device plugins must match the node image. A version skew that would be a nuisance on a web fleet becomes a silent training failure on a GPU fleet.
In practice, teams that treat AI infrastructure like a slightly bigger web app hit these walls within weeks.
Kubernetes as the control plane for AI
Kubernetes, open-sourced by Google in June 2014 and reaching v1.0 in July 2015, has become the common substrate for AI workloads. It gives teams one declarative API for scheduling, scaling and rolling out changes.
GPU scheduling depends on the device plugin mechanism, which reached stable status in Kubernetes 1.10 in March 2018. The Kubernetes documentation on device plugins explains how GPUs and other hardware get advertised to the scheduler. Without a working device plugin, pods request GPUs and simply never start.
A few practical notes from teams running this at scale:
- Pin the device plugin version in the same repository as the node pool definition. They change together.
- Use node taints and tolerations so general workloads cannot occupy GPU nodes.
- Separate training and inference node pools. Their scaling behavior and cost profiles differ.
- Keep the container runtime and driver versions in a single variables file, not scattered across manifests.
Kubernetes also makes the AI stack visible in one place. Model servers, autoscalers and batch schedulers all become YAML objects that a reviewer can read.
GitOps: Git as the source of truth
The GitOps pattern, popularized by Weaveworks around 2017, treats Git as the single source of truth. An agent inside the cluster continuously reconciles live state with what the repository declares. The OpenGitOps principles set out the canonical four-part definition: declarative, versioned and immutable, pulled automatically, and continuously reconciled.
For AI platforms, GitOps solves a specific pain. GPU clusters are often changed by hand during incidents, and those changes vanish from memory. With reconciliation, a manual hotfix either gets committed or gets reverted. Both outcomes are better than drift.
The CNCF accepted Argo in 2020 and Flux in 2022 as graduated projects, so both have mature ecosystems. Choosing between them matters less than committing to one and enforcing it. Teams that run two reconciliation tools against the same cluster usually regret it.
There is a useful bridge here from traditional operations. Many of the reliability habits that make cloud platforms stable transfer directly, as this piece on how cloud reliability skills transfer to AI infrastructure describes.
Policy as code: guardrails for expensive mistakes
Policy as code means writing rules that are evaluated automatically against infrastructure plans and live resources. Open Policy Agent, created at Styra and donated to the CNCF, graduated in 2021. Its Rego language is commonly used to enforce guardrails before a plan is applied.
The Open Policy Agent documentation covers how policies are written and evaluated. For AI platforms, a small set of rules prevents most expensive incidents:
| Rule | What it blocks | Typical trigger |
|---|---|---|
| Approved instance families only | Unsupported or overpriced GPU types | Plan review |
| No public storage buckets | Leaked datasets or weights | Plan review |
| Mandatory cost tags | Untracked spend | Plan review |
| Signed images only | Tampered containers | Admission control |
| Namespace resource quotas | Runaway jobs | Admission control |
The pattern that works is to start with three or four rules that catch real incidents, then add more only when something goes wrong. Teams that begin with forty policies find themselves granting exemptions within a month, and the whole system loses credibility.
Model artifacts as infrastructure
One of the more interesting shifts is treating model deployments as declared infrastructure. Tools such as KServe and the Kubeflow model-serving components let teams describe a model endpoint in YAML alongside the rest of the stack. KServe became a CNCF incubating project in 2023.
That means a model rollout can follow the same review path as a database migration. The manifest names the model artifact, the serving runtime, the resource requests and the rollout strategy. Rollback is a revert.
Supply-chain integrity fits here too. Sigstore’s cosign signs container images, and SLSA, a supply-chain framework that reached v1.0 in 2023, defines levels of build provenance. The SLSA v1.0 specification explains those levels in detail. Applying the same signing and provenance checks to model weights is newer territory, but the logic holds: if you would not run an unsigned container, you should not serve an unsigned model.
A practical build order that works
Teams often ask where to start. The sequence below reflects what tends to survive contact with real deadlines.
- Codify the network and cluster skeleton first. VPC, subnets, node pools and the Kubernetes distribution. Nothing AI-specific yet.
- Add the GPU node pool with pinned driver and plugin versions. Test that a trivial pod can request a GPU and run.
- Put the repository under GitOps reconciliation. From this point, manual changes are visible.
- Add three policy guardrails. Instance families, public buckets, cost tags.
- Declare the model serving layer. One endpoint, one manifest, one rollback path.
- Add artifact signing and provenance checks. Container images first, model weights later.
Each step is independently useful. That matters, because AI platform projects rarely get to finish everything before the first real workload arrives.
Common mistakes and what actually works
A handful of failure modes recur across teams.
Mixing Terraform and Kubernetes responsibilities. Terraform provisions the cluster; it should not manage every workload object inside it. The split that works is Terraform for infrastructure and GitOps for workloads. Blurring the line produces state files that take twenty minutes to plan.
Ignoring the cost of a plan. A plan that adds fifty GPU nodes looks fine in a pull request. Attaching an estimated monthly cost to the plan output changes review behavior immediately.
Treating GPU capacity as always available. Fallback regions and spot instance policies belong in the code, not in a runbook nobody reads.
Skipping the local test loop. Being able to run the same manifests against a small local cluster catches most typos before CI does.
Letting secrets into the repository. This is the oldest mistake in the book and still the most common. Use a secret manager and reference values, never literal keys.
For teams building the surrounding documentation and runbooks, there is a useful parallel in how technical writing supports enterprise infrastructure teams. The same discipline that makes a runbook readable makes a repository navigable.
How AI infrastructure as code differs from general IaC
The table below summarizes where the AI version departs from the general practice.
| Dimension | General IaC | AI infrastructure as code |
|---|---|---|
| Resource type | Mostly fungible compute | Scarce, expensive accelerators |
| State | Largely disposable | Checkpoints and weights persist |
| Version coupling | Loose | Drivers, CUDA, plugins must match |
| Cost profile | Predictable per hour | Spiky and capacity-constrained |
| Artifact trust | Container images | Images plus model weights |
| Scaling trigger | Request volume | Queue depth and job backlog |
None of these differences require new tools. They require different defaults, different guardrails, and a sharper focus on what happens when capacity runs out. For a wider view of how these layers fit together, the cloud infrastructure and architecture section covers the underlying patterns.
There is also a skills angle. Operating accelerator fleets demands familiarity with scheduling, drivers and capacity planning, which is why teams increasingly look for skills that support accelerator infrastructure operations when hiring. Writing the stack as code makes those skills transferable rather than personal.
Where the practice is heading
Vendor commentary in 2025 and 2026 frequently claims that AI agents will write infrastructure code, replacing hand-authored modules. That claim is worth treating as opinion rather than fact. No widely cited, peer-reviewed study yet quantifies adoption, and the underlying problem, deciding what the infrastructure should look like, does not disappear when a model drafts the YAML.
What is more defensible is that generation and review are separating. Agents can draft modules and propose changes; humans still approve plans and own the guardrails. The interesting question for platform teams is not whether a model can write a Terraform module. It is whether the review process can catch a bad one.
A second open question is tooling fragmentation. Terraform’s August 2023 license change to the Business Source License prompted the creation of the OpenTofu fork, and market-share figures between the two remain contested. Cite a dated survey rather than a vendor blog if you need numbers. The practical advice is to keep modules portable and avoid provider-specific features you cannot replace.
Conclusion
AI infrastructure as code is less a new tool category than a set of disciplines applied to unusually demanding hardware. The pieces are well established: Terraform or a fork for provisioning, Kubernetes with device plugins for scheduling, GitOps for reconciliation, policy as code for guardrails, and signed artifacts for trust. What changes is the cost of getting it wrong. A misconfigured web service wastes minutes; a misconfigured GPU fleet wastes days and a large invoice.
The teams that succeed tend to start small, codify the cluster skeleton before the model layer, and add guardrails only when a real incident justifies them. They treat the repository as the source of truth and manual changes as temporary. That approach will not make GPU capacity appear, but it will make every rebuild predictable, reviewable and cheaper than the last one.
Frequently asked questions
What is AI infrastructure as code?
It is the practice of defining the compute, networking, storage and model-serving layers of a machine learning platform in versioned files rather than manual configuration. The same tools used for general cloud infrastructure apply, extended to GPU pools, device plugins and model artifacts.
Is Terraform required for AI infrastructure as code?
No. Terraform is popular because its provider model covers most clouds and Kubernetes, but Pulumi, Crossplane and cloud-native tools are viable. What matters is that the definition lives in version control and can be reviewed and rebuilt.
How do GPUs get scheduled in Kubernetes?
Through the device plugin mechanism, which advertises hardware resources to the scheduler. A pod requests a GPU in its resource specification, and the scheduler places it on a node that has one available. Without a working plugin, those requests never get satisfied.
What does policy as code protect against in AI platforms?
Mostly expensive mistakes. Typical rules block unapproved GPU instance types, public storage buckets, unsigned images and missing cost tags. Policies are evaluated against plans before they are applied and against live resources through admission control.
Can model deployments really be managed as infrastructure?
Yes, and increasingly they are. Tools such as KServe let teams declare a model endpoint, its runtime and its resource requests in YAML. Rollouts then follow the same review and rollback path as any other infrastructure change.
Do AI agents replace the need for infrastructure as code?
Not yet, and probably not in the way vendors suggest. Agents can draft modules and propose changes, but someone still has to approve plans, own the guardrails and decide what the infrastructure should look like. The review process becomes more important, not less.
