Why Your Pod Can't Reach: Kubernetes Service, Endpoints, and Ingress

"Pod can't reach the service" is the Kubernetes equivalent of a printer problem — everyone has an opinion, almost nobody knows which of four objects is at fault.

Each has one job, and the failure is almost always in exactly one of them.

The four objects, and which one does what

The Service never forwards a packet. It's a promise. Something else — kube-proxy writing iptables/nftables rules, or a CNI implementing it — makes the promise real.

ClusterIP is a virtual IP in every node's routing rules

kube-proxy programs rules on every node so connecting to that virtual IP load-balances across the ready pod IPs from the Endpoints object.

Two consequences people trip on:

• The virtual IP is not routable from outside the cluster. Not a bug; that's the point.
• It works without DNS. You can curl the IP directly from inside. So if the IP works and the name doesn't, you have a DNS problem, not a networking problem — that single test splits most investigations in half.

Endpoints are where the magic isn't

When a Service has no endpoints, connections get refused or black-holed. This happens constantly:

kubectl get endpoints my-service
# or, on newer clusters:
kubectl get endpointslice -l kubernetes.io/service-name=my-service

Empty output means one of three things, in this order:

1. No pods match the selector. Label mismatch. kubectl get pods --show-labels.
2. Pods exist but aren't Ready. Failing readiness probe, or the probe hasn't fired yet. The endpoint only appears when Ready goes true.
3. Wrong namespace. Endpoints are namespaced; Service and pods must agree.

Almost every "service is broken" ticket where pods are visibly running is #2 — a readiness probe that never succeeds, or one pointing at a path that doesn't exist. The pod looks healthy in get pods because Running only means the container started.

Ingress is an HTTP router, not a load balancer

Ingress matches on Host header and path, then forwards to a Service. It doesn't know about TCP unless you use a TCP-capable controller, and it terminates nothing itself unless TLS is configured.

rules:
  - host: api.example.com
    http:
      paths:
        - path: /v2
          pathType: Prefix
          backend: { service: { name: api-v2, port: { number: 80 } } }

Failures here are almost always: the host doesn't match what the client sent, pathType is Exact when you wanted Prefix, or the controller never created a load balancer. kubectl describe ingress shows the events plainly.

Where DNS misleads you

Two specific ways:

• Negative caching. A lookup for a name that doesn't exist gets cached, so you keep getting NXDOMAIN after fixing the typo. Wait it out.
• The search path. Inside a pod, a bare name like api expands through namespace.svc.cluster.local. It resolves in one namespace and not another — "works from the pod, fails from the debug container" is a namespace mismatch, not flakiness.

Also: a Service resolves to a virtual IP even with zero endpoints. DNS will hand you an address nothing is listening on.

The debugging order

1. Does the pod IP respond? (bypass the Service entirely)
2. Does the virtual IP respond? (rules programmed?)
3. Does the name resolve, and to what? (DNS)
4. Is the endpoint list correct? (selector / readiness / namespace)
5. Only then look at Ingress.

Steps 1–4 are two or five commands each and take under a minute. Skipping to step 5 is how people spend an afternoon in YAML.

One note: if pods can't reach each other, that's usually network policy rather than Services — a default-deny with nothing opened, exactly as in the Kubernetes security baseline. The node/VPC layer beneath all this is AWS networking, which behaves on its own rules entirely.

Summary

Service declares, Endpoints resolve, CoreDNS names, Ingress routes — four objects, four jobs. Test from the bottom up (pod IP → virtual IP → DNS → endpoints → ingress), and remember that a Service with no ready endpoints resolves perfectly and connects nowhere. Most "Kubernetes networking is broken" reports are one kubectl get endpoints away from an answer.