ClusterIP, NodePort, LoadBalancer: What You're Actually Buying
It was a Friday afternoon, maybe 3:15, when a finance person forwarded me a cloud invoice with one line highlighted: "Load balancers: 23 active." We had eleven microservices. She wanted to know, politely, why we were paying for more than two load balancers per service, and I did not have a good answer ready.
The answer, it turned out, was that every single internal service had been created with type: LoadBalancer because someone copy-pasted a tutorial manifest in month one and it became house style. Twenty-three cloud load balancers, most of which existed to serve traffic that never left the VPC. The fix took an afternoon. The lesson took longer: most engineers I work with can recite the three service types, but very few can tell you what each one actually creates on the wire.
Quick answer: A Kubernetes Service gives a set of pods a stable virtual IP and DNS name. ClusterIP (the default) is reachable only inside the cluster and covers the vast majority of traffic. NodePort additionally opens a port in the 30000-32767 range on every node's IP so external systems can reach the service. LoadBalancer additionally asks your cloud provider to provision a real, billed load balancer pointing at those node ports. Use ClusterIP for everything internal, front external traffic with one ingress or gateway, and reserve type LoadBalancer for the handful of entry points that genuinely need it.
What a Service actually is (and isn't)
A Service doesn't run anywhere. There's no service process, no proxy pod, no magic box. A Service is a record: a name, a virtual IP allocated from the service CIDR, and a set of selector-matched endpoints. You can see the machinery directly:
$ kubectl get svc web -n shop
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web ClusterIP 10.96.182.14 <none> 8080/TCP 40d
$ kubectl get endpoints web -n shop
NAME ENDPOINTS AGE
web 10.244.1.37:8080,10.244.2.51:8080,10.244.3.12:8080 40dThe ClusterIP 10.96.182.14 exists only as a routing rule. No interface owns it, no host answers ARP for it. When a pod sends a packet to that IP, something on the node intercepts it and rewrites the destination to one of those three endpoint IPs. That something is kube-proxy.
Two facts fall out of this that people get wrong constantly. First, the ClusterIP is stable but the endpoints churn -- kill a pod and the endpoints object updates within a second or two, and the routing follows. That's the entire point: your app talks to a name that never changes while the pods behind it come and go. Second, a Service with no matching pods is a black hole. The record exists, DNS resolves, connections just time out. "Service exists" and "service works" are different statements, and kubectl get endpoints is how you tell them apart.
kube-proxy: the thing doing the actual work
Every node runs kube-proxy, and kube-proxy's whole job is programming the node so that packets to service IPs get delivered to endpoints. There are three modes you'll encounter.
iptables mode is the default almost everywhere. kube-proxy writes iptables rules that match on the ClusterIP and DNAT the packet to a randomly chosen endpoint. You can watch the rule set on any node:
$ sudo iptables-save -t nat | grep -A3 'KUBE-SVC.*web'
-A KUBE-SERVICES -d 10.96.182.14/32 -p tcp -m tcp --dport 8080 -j KUBE-SVC-X7Q2
-A KUBE-SVC-X7Q2 -m statistic --mode random --probability 0.333 -j KUBE-SEP-A1B2
-A KUBE-SVC-X7Q2 -m statistic --mode random --probability 0.500 -j KUBE-SEP-C3D4
-A KUBE-SVC-X7Q2 -j KUBE-SEP-E5F6Notice the --probability entries: load balancing is done by random rules, evaluated in order, per connection. It's dumb and it works. The catch is that iptables is a linked list. Five thousand services means tens of thousands of rules that every packet walks, and rule updates get slow. That's where IPVS mode comes in: same idea, but destinations live in a kernel hash table with real scheduling algorithms, so lookups are O(1) and you can pick round-robin or least-connection instead of random. If you're past a couple thousand services, switch.
The third option is no kube-proxy at all, which is where eBPF-based CNIs like Cilium are taking the world -- the service translation happens in eBPF programs at the socket layer instead of in iptables at all. That's a post of its own, but if you're running Cilium with kube-proxy replacement, everything in this section still describes the semantics, just not the implementation.
ClusterIP: the default for a reason
ClusterIP is what ~90% of your services should be. Internal DNS (web.shop.svc.cluster.local), a stable virtual IP, load balancing across endpoints, and zero exposure outside the cluster. It's also the only type that's free of footguns at any scale.
The one variant worth knowing is the headless service: set clusterIP: None and you skip the virtual IP entirely. DNS then returns the individual pod IPs directly instead of one stable address. That's what StatefulSets use, because databases and queues often want clients to address specific pods (postgres-0.postgres.shop.svc.cluster.local) rather than any-random-pod. If you've ever wondered why a StatefulSet demands a serviceName, this is why.
NodePort: every node, one port, no mercy
NodePort takes a ClusterIP service and adds one thing: kube-proxy opens the same port -- allocated from 30000-32767 unless you've changed the range -- on the IP of *every* node, forwarding to the service endpoints.
apiVersion: v1
kind: Service
metadata:
name: web
namespace: shop
spec:
type: NodePort
selector:
app: web
ports:
- port: 8080
targetPort: 8080
nodePort: 31080Now http://<any-node-ip>:31080 reaches the service, regardless of which node the pods are actually on, because kube-proxy will happily forward cross-node. That's the correct mental model: NodePort isn't "expose the pod," it's "the whole cluster answers on this port."
NodePort is the right answer in exactly two situations. First, when you bring your own load balancer -- on bare metal, or when your corporate F5/HAProxy needs a stable backend port per service. Second, for ten minutes of debugging. It is not a public API surface. Users should never see a URL with :31080 in it, and if you're pasting node IPs into anything that isn't an LB backend pool, you're building on sand -- nodes get replaced, their IPs change, and nothing health-checks them for you.
One knob matters enormously here: externalTrafficPolicy. The default, Cluster, means a node that receives traffic may forward it to a pod on a different node, which requires SNAT and destroys the client source IP. Set externalTrafficPolicy: Local and the node only forwards to local pods, preserving the source IP but risking imbalance if pods aren't spread evenly. Neither is wrong. Pick per service, deliberately.
LoadBalancer: you're buying infrastructure, one per service
type: LoadBalancer is NodePort plus a phone call to the cloud API. The cloud controller manager sees the service, provisions a real load balancer in your account, points it at the node ports, and writes the external hostname back into the service status:
$ kubectl get svc web -n shop
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
web LoadBalancer 10.96.182.14 a1b2c3-1234567890.elb.amazonaws.com 443:31080/TCPThat EXTERNAL-IP is a line item. On AWS it's roughly $16-25/month per NLB before traffic; the other clouds are in the same ballpark. Multiply by every service someone copy-pasted and you get my Friday afternoon with finance.
The correct architecture, almost always: one cloud load balancer in front of an ingress controller or Gateway API implementation, and ClusterIP services behind it. You get TLS termination, host- and path-based routing, and one line item. Reserve type: LoadBalancer for things ingress can't do -- raw TCP, UDP game servers, non-HTTP protocols -- or for the ingress controller's own service itself.
Useful annotations if you do use them: service.beta.kubernetes.io/aws-load-balancer-scheme: internal keeps the LB inside the VPC, and on most clouds you can pin the load-balancer source ranges with spec.loadBalancerSourceRanges so the whole internet can't poke your node ports. On-prem, MetalLB fills the same role by handing out IPs from a pool and advertising them over BGP or ARP.
Frequently asked questions
What is the difference between ClusterIP, NodePort, and LoadBalancer?
ClusterIP gives your pods a stable virtual IP that's only reachable inside the cluster. NodePort builds on ClusterIP and additionally opens the same high port (30000-32767 by default) on every node's IP, so external traffic can reach the service through any node. LoadBalancer builds on NodePort and asks the cloud provider to provision a real external load balancer that forwards to those node ports. Each type is a superset of the previous one, not a separate mechanism.
When should I use NodePort instead of LoadBalancer?
Use NodePort when you already have your own load balancer or proxy in front of the cluster and just need a stable, well-known port to point it at, which is common on bare metal and in cost-sensitive setups. It's also handy for quick debugging. What you should not do is hand out node IPs with random 30000-range ports to end users, because that's fragile, ugly, and bypasses health checking at the load balancer layer.
Why does every LoadBalancer service cost money on cloud Kubernetes?
Because the cloud controller manager provisions a real load balancer resource in your cloud account for each Service of type LoadBalancer, and those are billed per load balancer plus traffic. Ten LoadBalancer services on AWS means ten NLBs or CLBs on your invoice. The usual fix is to consolidate behind a single ingress controller, so one cloud load balancer fronts many services.
What does externalTrafficPolicy: Local do?
It stops kube-proxy from forwarding incoming external traffic to pods on other nodes, so the client's source IP is preserved and you skip one hop of NAT. The trade-off is that a node only serves traffic if it has a healthy local endpoint, which can unbalance load across nodes, and health checks against nodes without endpoints will fail. Set it when source IP matters for logging, rate limiting, or compliance, and you have an LB health check that understands it.
Is kube-proxy iptables mode a performance problem?
Only at real scale. iptables rules are evaluated in order, and the rule list grows linearly with the number of services, so clusters with many thousands of services start to feel it in latency and rule-update time. IPVS mode uses a hash table instead and handles large service counts better. For a few hundred services, iptables mode is fine and you have bigger problems to worry about.
The practical version
The three service types aren't three different products. They're one mechanism -- a virtual IP and a list of endpoints, programmed into every node by kube-proxy -- with two progressively more expensive ways of letting the outside world in. ClusterIP is the workhorse, NodePort is a building block for your own load balancing, and LoadBalancer is a purchase order.
Learn to read kubectl get endpoints and the iptables rules on a node, and services stop being magic. Learn that type: LoadBalancer has a unit price, and finance stops emailing you on Fridays.