conndeck Download

conndeck blog

Kubernetes Network Policies: Start With Default Deny

#kubernetes#networking#security#network-policy

The first default-deny policy I ever applied took down the cluster's DNS in under a minute, and I consider that one of the most valuable minutes of my career. Nothing teaches you how NetworkPolicies actually work like watching every pod in a namespace fail name resolution simultaneously, because your carefully written allow rules forgot that applications resolve names before they connect.

Since then I've rolled out network segmentation in enough clusters to have a fixed order of operations, and it starts from the opposite end than most teams expect: you don't write allow rules first. You write the deny, watch what screams, and earn your allowlist from evidence.

Quick answer: Apply a default-deny pair (empty podSelector, no ingress rules; same for egress) per namespace, then allow only observed flows, starting with DNS. The two traps that break rollouts: a CNI that doesn't enforce policies at all (your YAML silently does nothing), and forgotten DNS egress, which makes the deny look like an app outage. Stage it with audit mode where available.

The additive rule that explains everything

NetworkPolicy semantics confuse people because they're unusual: policies are additive, never subtractive, and they only bite when they select a pod.

  • A pod selected by zero policies: all traffic allowed, both directions.
  • A pod selected by any policy's podSelector for ingress: ingress is now deny-by-default, except where any policy explicitly allows it.
  • Same, separately, for egress.

That's why the default-deny primitive is almost embarrassingly small:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}          # every pod in this namespace
  policyTypes: ["Ingress", "Egress"]
  # no rules = deny everything, both directions

Empty podSelector: {} selects all pods in the namespace; the missing rule lists are what denies. Every connection your workloads ever need now has to be earned back with an allow rule, which is exactly the point, and exactly the outage risk if you do it to a running namespace in one shot.

The CNI silence trap

The most dangerous thing about NetworkPolicies is what happens when nobody enforces them: nothing. No error, no event, no warning. The API server happily stores your policy, kubectl get networkpolicy shows it, and traffic flows exactly as if it didn't exist.

The stock Kubernetes distribution ships the policy API but no enforcement -- that's CNI territory. Cilium, Calico, and Antrea enforce. Flannel doesn't. Some managed setups surprise you too: certain cluster installations "support" the API while the dataplane ignores it.

So step zero of any network policy effort is an enforcement test that takes two minutes:

kubectl run a --image=busybox --rm -it --restart=Never -- wget -qO- --timeout=2 http://b.default:8080

Two throwaway pods, a default-deny between them, and a curl that must now fail. If it succeeds, stop everything and fix your CNI before writing a single allowlist rule, because you'd be documenting a security posture you don't have. This is the same "trust but verify the enforcement layer" instinct as checking what apply actually did rather than assuming the manifest ran.

The DNS rule every namespace needs

Back to my minute-long outage. Default deny went on, and the first casualty wasn't an application connection -- it was resolution. Applications couldn't even *find* the services they were allowed to talk to, because CoreDNS sat outside the allowlist.

The universal allowance, adjusted for your DNS labels:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: payments
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Note the namespaceSelector plus podSelector in the same to: entry: that's an AND (pods with this label, in namespaces with that label). Put them as separate list items and it becomes an OR, which is a completely different (and sloppier) allow. This AND/OR switcheroi based on whether selectors share an element bites everyone once.

If DNS still misbehaves under load after this, don't blame the policy by default; the usual CoreDNS failure patterns apply on top of segmentation, not because of it.

The allowlist I end up writing, in order

After DNS, my allow rules have converged to a predictable sequence, which makes rollouts reviewable:

  1. Intra-namespace app traffic. Allow the namespace's own pods to reach each other on declared ports. For most teams this single rule plus DNS unblocks 80% of the namespace.
  2. Ingress from the edge. Whatever fronts the namespace (ingress controller pods, gateway) gets explicit ingress by namespace and label.
  3. Known external dependencies. Databases, caches, queues -- by ipBlock with an except for ranges you specifically don't mean, or by label if they're in-cluster.
  4. Egress to the internet, if the namespace needs it at all. This one deserves friction: most namespaces think they need egress and are wrong. Restricting it is where you catch the surprise exfil paths and the dependency on a third-party API nobody documented.

Two operational notes from scar tissue:

  • Kubelet probes originate from the node, not from the pod. With strict ingress deny, health endpoints can get caught in the net depending on your CNI's handling; Cilium and Calico generally special-case this, but verify rather than assume, because the failure shows up as flapping readiness across a whole namespace.
  • Watch pod-to-pod allow rules rot. Every allow rule references labels, and labels drift. A quarterly review of "allows that no longer match anything" keeps the policy set honest. This is fleet-level housekeeping, which is precisely why I keep cluster-wide views around (multi-cluster drift is the same disease at a bigger scale).

Staging the rollout without an outage

Deny-by-default on a busy namespace is a change of state for every connection in it, so I treat it like a schema migration:

  1. Audit first. Cilium has policy audit mode; Calico has similar tiers/log-only options. Run default deny in log-everything mode for a week and collect the flows.
  2. Allowlist from evidence, not from architecture diagrams. The logs will show you flows the diagram never mentioned -- the metrics scraper, the backup job, the sidecar that phones home.
  3. One namespace at a time, starting with the least critical, with the owner team watching.
  4. Flip to enforce, keep logging on. Denied-flow logs after enforcement are your tripwire for the dependency nobody documented.

This staging discipline is the practical answer to "isn't default deny too risky?" The risk isn't the policy; it's applying a global behavior change without observation. Audit mode exists precisely so that the first enforcement moment is boring.

Where this fits in the security baseline

Default deny is the network half of a zero-trust posture inside the cluster; the identity half is RBAC and workload identity, and neither substitutes for the other. If you're building out the rest of the baseline, the pod security standards are the compute-side companion piece, and the security pages document how we think about local-first posture more broadly.

The honest caveat about scope: NetworkPolicies operate on L3/L4 -- IPs, ports, protocols. They cannot express "this pod may call GET /orders but not DELETE /orders." That's a service-mesh or authorization-layer concern. Don't let anyone sell a mesh as a prerequisite for segmentation, though; deny-by-default L3/L4 closes most of the practical attack surface in a typical cluster.

Frequently asked questions

Do Kubernetes NetworkPolicies require a specific CNI?

Yes. The core Kubernetes distribution ships the NetworkPolicy API but no enforcement. Flannel, for example, ignores policies entirely, so your default-deny YAML applies as an object and does absolutely nothing. You need a policy-capable CNI such as Cilium, Calico, or Antrea. Always test enforcement with two throwaway pods before trusting any policy.

How do I allow DNS after applying a default-deny policy?

Add an egress policy that targets the DNS pods by namespace and label, typically namespaceSelector matching kube-system plus podSelector matching k8s-app: kube-dns, and allow UDP and TCP on port 53. Missing this is the single most common reason a fresh default-deny rollout looks like an application outage.

Why does traffic still flow to a pod that has a NetworkPolicy?

Policies are additive: a pod is only isolated on a path once at least one policy selects it for that direction. A policy with an empty podSelector and no ingress rules selects every pod in the namespace, which is how you write default deny. If traffic still flows, the likely causes are a podSelector typo, the policy living in the wrong namespace, or a CNI without enforcement.

Is default deny too risky to enable in a running cluster?

It is risky to enable blind, which is why I stage it. Start in audit mode if your CNI supports it, log what would be denied, ship the allowlist for observed flows, then flip to enforcement namespace by namespace. Clusters that do this get a real security posture; clusters that apply deny-all on a Friday get an outage and a rollback.

Start with default deny

The full segmentation policy set matters less than the posture: deny by default, allow from evidence, verify enforcement, stage the rollout. A cluster with an imperfect allowlist behind default deny is safer than a cluster with a beautiful policy nobody has tested, and both are safer than a cluster whose policies aren't enforced at all.

And make the current state visible. Half the network-policy incidents I've watched came from someone changing labels or namespaces and not knowing which allows went dark. A view that shows which policies select which pods, per cluster, turns that guesswork into a lookup -- it's one of the things we built conndeck's fleet views around, and it's the difference between auditing segmentation and hoping about it.