What do you do when a pod that worked fine yesterday refuses to start, and the only clue is a status message that sounds like a spell from a fantasy novel? Most Kubernetes teams hit this wall eventually. The platform is powerful, but it hides its reasoning behind short status strings and layers of abstraction. The good news is that the same handful of failures account for the vast majority of incidents. Learn to recognize them and you will spend far less time guessing. (Kubernetes Documentation: Overview of Kubernetes Concepts)
This guide walks through the problems engineers actually run into, what each one looks like, why it happens, and how to fix it. It also covers the habits that separate teams who recover in minutes from teams who debug for hours.
Why Kubernetes Problems Repeat So Predictably
Kubernetes is a declarative system. You describe the state you want, and controllers work to make reality match. When something breaks, it is almost always because the description was wrong, the environment could not satisfy it, or a resource ran out. That is a short list of root causes, which is why the same symptoms keep showing up across completely different organizations.
The trick is reading the signals correctly. A pod status is a symptom, not a diagnosis. The real story lives in events, logs, and the container’s exit code. Engineers who internalize that distinction resolve incidents faster than those who memorize commands.
CrashLoopBackOff: The Most Common Kubernetes Problem
CrashLoopBackOff means a container starts, exits, and gets restarted on a growing delay. The backoff doubles each time up to a five-minute cap. People often treat the status as the problem, but it is really the scheduler saying “I keep trying and it keeps dying.”
Common causes include a missing environment variable, a bad command or entrypoint, a failing liveness probe that kills a healthy-but-slow app, or an application that exits cleanly because its main process runs in the background. That last one trips up newcomers constantly: if the container’s foreground process finishes, the container is done, no matter what else is running.
How to debug it
- Run `kubectl describe pod ` and read the Events section at the bottom.
- Check `kubectl logs –previous` to see output from the container that died.
- Confirm the image’s entrypoint actually stays in the foreground.
- Loosen liveness probes with a generous `initialDelaySeconds` and `failureThreshold` for slow-starting apps.
A frequent mistake is setting the liveness probe to match the readiness probe. Readiness should fail fast so traffic stops routing; liveness should be forgiving so the app is not killed mid-startup.
OOMKilled and Memory Limit Mistakes
Exit code 137 usually means the kernel’s out-of-memory killer terminated the container for exceeding its memory limit. The pod restarts, hits the same ceiling, and the cycle looks exactly like a crash loop.
The classic culprit is a runtime that does not know about the container limit. A JVM sized for the host’s memory will happily allocate past a 512Mi limit and get killed. Modern JVM versions respect container awareness, but many teams still hardcode heap flags that ignore it.
Practical fixes:
- Set memory requests equal to limits for critical workloads so they land in the Guaranteed QoS class.
- Size heap at roughly 70 to 80 percent of the container limit, leaving room for off-heap memory.
- Watch actual usage with metrics before adjusting, rather than guessing upward.
Overprovisioning limits “just in case” has a cost too. It wastes capacity and can make scheduling harder across a busy cluster. The right number comes from observation, not fear.
ImagePullBackOff and ErrImagePull
These two statuses mean the kubelet could not fetch the image. The causes are boring but numerous: a typo in the image name, a tag that does not exist, a private registry without imagePullSecrets, or a registry rate limit on anonymous pulls.
Docker Hub’s anonymous pull limits, introduced in late 2020 and revised several times since, are a commonly cited trigger in shared clusters where many pods pull the same public image. The durable fix is to mirror images into your own registry or authenticate pulls.
A subtle variant: the image pulls fine on one node but not another. That usually points to a node-level credential problem or a node in a different network zone that cannot reach the registry. Checking which node the pod landed on is a step people skip.
Pending Pods and Scheduling Failures
A pod stuck in Pending has not been placed on any node. The scheduler is telling you it cannot find a match. Reasons cluster into a few buckets:
| Symptom in Events | Likely Cause | Fix |
|---|---|---|
| Insufficient cpu or memory | Requests exceed free capacity | Lower requests or add nodes |
| node(s) didn’t match node selector | Label mismatch | Correct selector or label the node |
| node(s) had untolerated taint | Missing toleration | Add toleration or remove taint |
| pod has unbound immediate PersistentVolumeClaims | No matching PV | Fix storage class or PVC size |
| 0/N nodes available | Affinity rules too strict | Relax affinity or anti-affinity |
The most underrated diagnostic here is the resource request. Teams often copy a manifest with 2 CPU requested per replica, then wonder why a ten-replica deployment sits half-pending. Requests are a scheduling contract, not a performance setting.
Service and Networking Problems
A deployment can be perfectly healthy and still unreachable. When a Service does not route traffic, the first thing to check is whether it actually selected any pods. The endpoints object (or EndpointSlice in modern clusters) lists the pod IPs behind a Service. If it is empty, the label selector does not match the pod labels, or the pods are not ready.
Other frequent causes:
- targetPort mismatch: the Service points at a port the container does not listen on.
- NetworkPolicy silently blocking traffic between namespaces.
- DNS resolution failures, often from a misconfigured CoreDNS or a pod using the wrong resolver.
A quick sanity check is to exec into another pod and curl the Service’s cluster IP directly. If that works but the hostname does not, the problem is DNS, not routing. Splitting the test that way saves a lot of aimless poking. (Blog)
Ingress Returns 404 or 503
Ingress issues frustrate people because the error comes from the ingress controller, not the app. A 404 typically means no rule matched the request: wrong host header, wrong path, or a missing IngressClass so the controller ignored the resource entirely. A 503 usually means the backend Service has no ready endpoints.
Check the controller logs first. They usually state plainly which rule was evaluated and why it failed. Also confirm the ingress controller is actually watching the namespace where your Ingress lives; some installations scope controllers by namespace, and resources elsewhere are simply invisible.
The Ingress API reached general availability in Kubernetes 1.19 in August 2020. The Gateway API is the longer-term successor for richer routing. Ingress is not deprecated, but new platforms increasingly evaluate that API’s resources for multi-team routing needs.
RBAC and Permission Errors
Forbidden errors are almost always a scope problem. A Role grants permissions inside one namespace; a ClusterRole applies cluster-wide. Binding a Role to a ServiceAccount in the wrong namespace, or granting a ClusterRole without a ClusterRoleBinding, produces the same confusing result.
The standard diagnostic is `kubectl auth can-i –as system:serviceaccount::`. It answers the question directly instead of making you reason through YAML. Run it before touching any manifest.
A common real-world trap: an application works locally with an admin kubeconfig, then fails in-cluster because its ServiceAccount has no permissions at all. Always test with the actual identity the workload uses.
Control Plane and etcd Health
Some problems are not about workloads at all. When the API server gets slow or requests start timing out, the control plane deserves attention. etcd is particularly sensitive to disk latency; slow fsync on the storage backing etcd shows up as elevated API latency and occasional leader elections. Monitoring etcd disk latency is one of the highest-value alerts a platform team can configure. (Wikipedia: Containerization (computing)))
Version skew is another quiet source of odd behavior. Kubelets are allowed to run somewhat older than the API server, but exceeding the supported range causes subtle failures. The official version skew policy page spells out the current allowances, and it changes between releases, so check it against your actual versions rather than relying on memory.
Practical Habits That Shorten Every Incident
Experience teaches a few habits that matter more than any single command.
Start with events, not logs. `kubectl get events –sort-by=.lastTimestamp` often names the problem outright. Read the exit code before reading application logs; 137 means memory, 1 means the app itself failed, 143 means a clean termination signal. Compare a broken pod’s spec against a working one from the same deployment; drift is easier to spot side by side. (Cloud)
Keep a runbook for the top ten symptoms, and keep cluster news and platform updates on your radar so you are not surprised by a behavior change. Following release notes and operational writeups, such as the platform coverage in the news section at 9481.org, helps teams catch breaking changes before they hit production. Deeper engineering notes live in the blog at 9481.org, and infrastructure patterns around managed clusters are covered under cloud at 9481.org. (News)
Finally, resist the urge to fix symptoms with restarts. A pod that recovers after deletion but fails again tomorrow has an unfixed root cause, and the next failure will probably happen at a worse time.
Conclusion
Common Kubernetes problems are common for a reason: they come from a small set of root causes, and the platform reports them with a small set of status strings. CrashLoopBackOff, OOMKilled, Pending, empty endpoints, and forbidden errors cover most of what teams actually face. Once you can map a symptom to its likely cause, troubleshooting stops feeling like guesswork.
The discipline that pays off is reading signals in order: status, then events, then exit code, then logs. Add a few habits, like checking endpoints before blaming the network and testing permissions with the real ServiceAccount, and incident time drops noticeably. Kubernetes will still surprise you occasionally, but the surprises get smaller and rarer.
Frequently Asked Questions
What does CrashLoopBackOff actually mean?
It means a container keeps exiting and the kubelet keeps restarting it, with a delay that doubles up to five minutes. The status describes the restart pattern, not the cause. You still need logs and events to find out why the container dies.
Why does my pod show OOMKilled even though memory looks fine?
The kernel kills a container the moment it crosses its memory limit, and monitoring often samples too slowly to catch the spike. Check the limit against peak usage, and remember that runtimes like the JVM may reserve memory beyond the heap. Exit code 137 is the confirmation.
Why is my Service not sending traffic to healthy pods?
The usual cause is a label selector that does not match the pod labels, or pods that are running but not ready. Check the endpoints or EndpointSlice for the Service. If it lists no pod IPs, the Service never selected anything.
What is the fastest way to debug a Pending pod?
Run `kubectl describe pod` and read the Events. The scheduler states its reason directly: insufficient resources, unmatched node selector, untolerated taint, or an unbound PersistentVolumeClaim. Each maps to a specific fix.
Why do I get Forbidden errors when my Role looks correct?
Scope is the likely issue. Roles only apply within their own namespace, and a ServiceAccount in a different namespace gets nothing. Use `kubectl auth can-i` with the `–as` flag to test the exact identity your workload uses.
Does a failing liveness probe really restart my container?
Yes. If the liveness probe fails enough times, the kubelet kills and restarts the container regardless of what the application is doing. For slow-starting apps, increase the initial delay and failure threshold so startup is not mistaken for a hang.
