What happens when a cluster that ran perfectly in staging starts dropping requests the moment real traffic arrives? In most cases, nothing exotic broke. Someone deployed a bare Pod instead of a Deployment, forgot to set memory limits, or skipped readiness probes. Kubernetes is forgiving enough to accept almost any manifest you hand it, which is exactly why so many clusters drift into a fragile state. The platform does what you tell it, not what you meant.
This guide covers the practices that consistently separate healthy clusters from painful ones. It focuses on decisions made early, because retrofitting resource limits, security policies, and upgrade habits onto a running production cluster is far harder than building them in from day one.
Start With the Right Workload Controller
The single most common beginner mistake is deploying bare Pods. A bare Pod is not rescheduled if its node fails, and it gets no rolling updates. For stateless applications, a Deployment is the right choice: it manages a ReplicaSet, performs declarative rolling updates, and replaces failed Pods automatically. The official documentation on how Deployments manage rolling updates and self-healing explains the reconciliation model in detail.
The practical rule of thumb:
| Workload type | Right controller | Why |
|---|---|---|
| Stateless web or API service | Deployment | Rolling updates, self-healing, easy scaling |
| Stateful database or queue | StatefulSet | Stable network identity, ordered rollout |
| One-off task or batch job | Job | Runs to completion, retries on failure |
| Scheduled task | CronJob | Time-based execution |
| Node-level agent (logging, networking) | DaemonSet | One Pod per node by design |
A second habit worth building: always run more than one replica for anything user-facing, and spread replicas across nodes with topology spread constraints or pod anti-affinity. A three-replica Deployment that lands all three Pods on one node gives you the illusion of redundancy and none of the reality.
Set Resource Requests and Limits Before Anything Else
If you do only one thing from this article, do this one. Resource requests tell the scheduler how much CPU and memory a Pod needs, which determines where it can be placed. Limits cap what the container can actually consume. The Kubernetes documentation on managing CPU and memory resources for containers covers how the scheduler and kubelet enforce both.
What goes wrong in practice:
- No requests at all. The scheduler treats the Pod as needing zero resources and packs nodes tightly. Under load, everything competes and latency spikes unpredictably.
- Limits equal to requests for CPU. This forces the CPU throttling behavior of the Guaranteed QoS class, which is fine for latency-sensitive services but wasteful for batch work.
- Memory limits set too low. The container gets OOM-killed under normal traffic peaks. Teams then raise the limit blindly instead of measuring actual usage.
- No limits on memory. One leaking container can consume a node and trigger evictions that take down unrelated workloads.
A workable approach is to measure first, then set requests close to typical usage and memory limits at roughly 1.5 to 2 times the observed peak. Tools like the Vertical Pod Autoscaler in recommendation mode can suggest values from real data, which beats guessing.
Get Probes Right, Including Startup Probes
Probes are where well-intentioned configuration causes outages. There are three types, and each does a different job, as described in the guide to configuring liveness, readiness, and startup probes.
Readiness probes decide whether a Pod receives traffic. Liveness probes restart a container that is stuck. Startup probes protect slow-starting applications from being killed by liveness checks before they finish booting.
The classic failure: a liveness probe with a short timeout on an app that takes 90 seconds to warm its cache. Kubernetes kills it, restarts it, kills it again, and the Pod enters a crash loop that looks like an application bug. A startup probe solves this cleanly. Startup probes became stable back in Kubernetes 1.20, so there is no reason to avoid them on any current cluster.
Two more practical notes. First, point readiness probes at a lightweight endpoint that checks dependencies, not at the main page. Second, never make a liveness probe depend on a downstream service. If your database blips, you do not want every replica restarting simultaneously.
Lock Down Access With Least-Privilege RBAC
Role-based access control determines who and what can do things in the cluster. The reference on RBAC roles, cluster roles, and bindings lays out the model. RBAC has been generally available since Kubernetes 1.8, so it is mature and well understood.
The most frequent mistake is binding workloads or developers to cluster-admin because it makes errors disappear. It also makes every compromised container a cluster-wide incident. A better pattern:
- Create a Role scoped to a namespace with only the verbs and resources actually needed.
- Bind service accounts to that Role, one service account per workload.
- Disable automounting of the service account token for Pods that never call the API server.
- Audit bindings regularly; stale cluster-admin grants accumulate quietly.
On managed platforms, cloud identity integrations can map external identities to Kubernetes permissions, which reduces the number of long-lived credentials floating around.
Enforce Pod Security and Network Segmentation
PodSecurityPolicy is gone. It was deprecated in Kubernetes 1.21 and removed in 1.25. Its replacement is Pod Security Admission, which enforces three levels (Privileged, Baseline, Restricted) through namespace labels. The overview of Pod Security Standards and how admission enforces them is the place to start.
In practice, most teams run the Restricted profile on application namespaces and Baseline on infrastructure namespaces. The transition is easier if you first run the policy in warn or audit mode, collect the violations, fix the manifests, then switch to enforce. Skipping that step is how teams end up blocking their own deployments at 2 a.m.
For traffic, NetworkPolicy lets you define which Pods may talk to which. The documentation on NetworkPolicy selection and ingress/egress rules notes an important caveat: enforcement depends on the CNI plugin. Some plugins ignore policies entirely, so a policy that looks correct may do nothing. Verify enforcement with a test connection before trusting it. A default-deny ingress policy per namespace, plus explicit allow rules, is the standard starting point.
Use Namespaces, Quotas, and LimitRanges Together
Namespaces do more than label objects. Combined with ResourceQuota and LimitRange, they form a governance layer. The guide to resource quotas and limit ranges explains how quotas cap total namespace consumption while LimitRanges set per-object defaults and bounds.
This combination catches the most common resource mistake: a developer deploys without requests or limits, and a LimitRange fills in sane defaults automatically. A quota then stops one team’s experiment from consuming the entire cluster. Without both, “we’ll add limits later” turns into a year of unbounded growth.
Plan Upgrades Instead of Reacting to Them
Kubernetes releases roughly three minor versions per year, and each minor version is supported for about 14 months, covering three minor releases. The official Kubernetes release cadence and support window is the authoritative source, and the exact schedule has shifted over time, so check it rather than relying on memory.
The practical advice: never let a cluster fall more than two minor versions behind. Upgrades are usually incremental and boring when done regularly, and stressful when skipped. Before each upgrade, check deprecated API usage with tools like kubectl deprecations or pluto, read the changelog for removed features, and test in a non-production cluster first. Managed control planes handle some of this for you, but node pools, admission webhooks, and custom resources still need attention.
Adopt Declarative Configuration and GitOps
Kubernetes is built around declarative configuration: you describe desired state, and controllers reconcile reality toward it. The explanation of how Kubernetes objects represent desired state is worth reading once, carefully, because it explains why manual kubectl edits keep getting reverted.
That model naturally leads to GitOps: manifests live in Git, a controller syncs them to the cluster, and every change is reviewable and reversible. Even without a GitOps tool, keeping manifests in version control with a clear promotion path from staging to production prevents the classic “who changed this and when” investigation.
Protect Availability During Maintenance
Node drains happen. Cluster upgrades, autoscaling, and hardware failures all move Pods around. A PodDisruptionBudget limits how many Pods of a workload can be voluntarily disrupted at once, which keeps a service available during drains. The documentation on Pod disruptions and disruption budgets covers the mechanics; PodDisruptionBudget has been stable since Kubernetes 1.21. (Wikipedia: Containerization (computing)))
A budget like minAvailable: 2 on a three-replica Deployment means drains proceed one Pod at a time. Without it, a maintenance operation can take every replica down at once. Pair budgets with readiness probes and multiple replicas, or they will not help much.
Watch the Cluster and the Apps
Observability is often treated as an application concern, but cluster-level signals matter just as much. Track node pressure conditions, pending Pods that never schedule, restart counts, and control plane latency. A Pod stuck in Pending almost always means insufficient requests, a missing toleration, or a taint mismatch, and it will never appear in application dashboards.
For teams running workloads across providers, keeping infrastructure notes and operational write-ups in one place helps, and resources like the cloud coverage on 9481.org can round out the picture. It also helps to follow industry news on 9481.org when planning upgrades, since release announcements and deprecation timelines are easy to miss. Teams that document their own runbooks often publish lessons in a technical blog on 9481.org, which is a reasonable model for capturing what actually broke.
Common Mistakes Worth Avoiding
A short list of habits that cause the most trouble:
- Copying a manifest from a blog post without checking its API version or resource settings.
- Running stateful databases on shared node pools with aggressive autoscaling.
- Using latest image tags, which makes rollbacks ambiguous.
- Skipping namespace-level quotas because “the cluster is big enough.”
- Granting cluster-admin to CI systems for convenience.
- Treating staging as optional, then discovering differences during a production incident.
None of these are exotic. They are ordinary shortcuts that compound.
Conclusion
Kubernetes best practices are less about obscure features and more about discipline. Deploy through controllers, set requests and limits, configure all three probe types, keep permissions narrow, enforce pod security and network policy, and upgrade on a schedule rather than in a panic. Each of these is documented, stable, and well supported. The teams that struggle are rarely missing knowledge; they are missing consistency. Pick two or three items from this list, apply them across every workload, and the cluster becomes noticeably calmer within a release cycle.
Frequently Asked Questions
What is the most important Kubernetes best practice for beginners?
Setting resource requests and limits on every container. It affects scheduling, stability, and cost more than any other single setting, and it is the easiest to overlook when copying example manifests.
Should I use Deployments or bare Pods?
Use Deployments for anything long-running. Bare Pods are not rescheduled after node failure and get no rolling updates. Bare Pods are only useful for quick debugging.
Do I need all three probe types?
Not always, but most production services benefit from readiness and liveness probes. Add a startup probe if the application takes more than about 30 seconds to become ready, which prevents premature restarts.
Is NetworkPolicy always enforced?
No. Enforcement depends on the CNI plugin installed in the cluster. Some plugins ignore NetworkPolicy objects entirely, so test connectivity after applying policies rather than assuming they work.
How often should I upgrade Kubernetes?
Aim to stay within two minor versions of current. Releases arrive roughly three times a year with about 14 months of support per version, so a regular cadence of one or two upgrades a year keeps you comfortably supported.
What replaced PodSecurityPolicy?
Pod Security Admission, which enforces the Privileged, Baseline, and Restricted Pod Security Standards through namespace labels. PodSecurityPolicy was removed in Kubernetes 1.25, so older manifests referencing it will fail on current clusters.
