kube-prometheus-stack: Monitoring That Won't Page You at 3AM
kube-prometheus-stack gets you 90% of the way there and pages you for the wrong 10%. How I tune alerts, add recording rules, and keep cardinality under control.
Read the postPosts tagged "kubernetes"
50 posts about kubernetes in the conndeck blog — field notes on local-first Kubernetes operations, GitOps, and production debugging.
kubectl debug: Ephemeral Containers Changed How I Troubleshoot
kubectl debug with ephemeral containers fixes the oldest problem in Kubernetes troubleshooting: distroless images with no shell. Distroless and node debugging.
Read the postKubernetes Cost Optimization: Finding the Idle Half of Your Cluster
Most clusters run 30-50% idle because requests don't match reality. A practical walkthrough of requests vs usage, OpenCost, right-sizing, and spot nodes.
Read the postSLOs for Kubernetes Services: Alerting on What Actually Matters
Stop paging on CPU and start paging on burn rate. A practical guide to SLOs, error budgets, and multi-window alerts for Kubernetes services.
Read the postUpgrading Kubernetes Without Downtime or Drama
Kubernetes upgrades don't have to be scary. Version skew rules, drain and PDB discipline, deprecation hunting, and how managed vs self-managed changes the job.
Read the postManaging a Fleet of Clusters Without Losing Track of Reality
From 2 clusters to 20: kubeconfig hygiene, fleet-wide visibility, and GitOps per cluster. The patterns that keep multi-cluster from becoming multi-chaos.
Read the postImage Scanning + Admission Control: Stop Vulnerable Images
How to scan container images with Trivy or Grype and enforce results with Kyverno or Gatekeeper admission policies, plus an exception workflow that won't get abused.
Read the postKubernetes Audit Logs: Finding the Needle Before the Incident
How to tune Kubernetes audit policy, what to actually alert on, and how to keep log volume from eating your budget. Practical audit logging for platform teams.
Read the postPod Security Standards: Getting to restricted Without a Revolt
A field-tested rollout plan for Pod Security Standards: audit mode first, baseline before restricted, and exemptions that don't become permanent.
Read the postRBAC Least Privilege: Designing Roles People Can Live With
Practical Kubernetes RBAC design: namespaced Roles over ClusterRoles, aggregated clusterroles for teams, killing wildcard verbs, and audits people actually run.
Read the postService Account Tokens: The Quiet Security Hole in Every Cluster
Service account tokens are mounted into every pod by default. Here's how automountServiceAccountToken, bound tokens, and workload identity close the hole.
Read the postSupply Chain Security: Sigstore, SLSA, and What's Actually Practical
A pragmatic take on supply chain security: cosign keyless signing, signature verification at admission, SBOMs that get used, and SLSA without ceremony.
Read the postetcd Backup and Restore: The Drill Nobody Practices
Everyone takes etcd snapshots; nobody restores one until it's life or death. Snapshots, quorum loss, and the restore procedure you should drill quarterly.
Read the postPVs, PVCs, and StorageClasses: The Mental Model That Finally Clicked
PVs, PVCs, and StorageClasses explained the way I wish someone had explained them to me: provisioning, reclaim policies, and expansion, minus the confusion.
Read the postRunning Postgres on Kubernetes: Operators Make It Sane
Postgres on Kubernetes doesn't have to be scary. How operators like CloudNativePG handle failover, backups, and upgrades, and why the objections are dated.
Read the postPVC Stuck in Pending: Every Reason and the Fix
A Pending PVC is always one of a few causes: missing provisioner, topology conflicts, WaitForFirstConsumer timing, or quota. Here's how to tell them apart.
Read the postStatefulSets: When You Actually Need One
StatefulSets give pods stable names, storage, and ordered rollout. When that matters, when a Deployment is fine, and the headless service nobody explains.
Read the postVelero: Backups That Survive a Real Disaster
Velero backups are easy to set up and easy to set up wrong. Schedules, kopia file-system backups, snapshot pitfalls, and the restore test that matters.
Read the postArgo Rollouts: Canary Deploys Without a Platform Team
A practitioner's guide to Argo Rollouts: canary steps, AnalysisTemplates with Prometheus metrics, aborts and rollbacks, and the gotchas that bite small teams.
Read the postArgoCD vs Flux: After Running Both in Production
An honest ArgoCD vs Flux comparison from someone who ran both in production: install, day-two ops, multi-tenancy, drift handling, and which one I'd pick when.
Read the postExternal Secrets Operator: Stop Committing Secrets
Why encrypted secrets in git still lose, and how to wire External Secrets Operator to Vault or a cloud secret manager with rotation that actually works.
Read the postPromoting Between Environments in GitOps Without Losing Your Mind
How to promote from staging to prod in GitOps: repo layouts, Kustomize overlays, image tag promotion via PR, and the habits that prevent drift.
Read the postHelm vs Kustomize: An Honest Take From Both Sides
Helm vs Kustomize without the tribalism: templating vs overlays, release management vs GitOps, drift behavior, and how I use both on the same cluster.
Read the postBuilding Images Inside Kubernetes: Kaniko vs BuildKit
Kaniko vs BuildKit for in-cluster image builds: why docker:dind is a dead end, how each tool actually works, real caching setups, and which one I'd pick.
Read the postWhy Your Cluster Autoscaler Isn't Scaling
Pods Pending, cluster autoscaler doing nothing? The usual causes: PDBs, node group limits, taints, labels, and resource requests no node can satisfy.
Read the postHPA, VPA, KEDA: Which Autoscaler Does What
HPA scales replicas on metrics, VPA right-sizes requests, KEDA scales on external events. How they differ, how they conflict, and how to combine them safely.
Read the postQoS Classes: Guaranteed, Burstable, and Who Gets Evicted First
Kubernetes assigns every pod a QoS class: Guaranteed, Burstable, or BestEffort. How the classes are computed and how they decide eviction order under node pressure.
Read the postLimitRange and ResourceQuota: Namespace Guardrails That Actually Work
LimitRange sets per-container defaults and bounds; ResourceQuota caps namespace totals. How they interact, the gotchas, and configs that hold up in production.
Read the postPodDisruptionBudgets: The Availability Feature Everyone Skips
PodDisruptionBudgets control how many pods voluntary disruptions can take down. Drain behavior, minAvailable vs maxUnavailable, and PDBs that block upgrades.
Read the postRequests and Limits: Stop Copying Numbers From Stack Overflow
Requests and limits drive scheduling, throttling, and OOM kills. A practical guide to measuring real usage, the CPU limits debate, and right-sizing with data.
Read the postPicking an Ingress Controller: nginx vs Traefik vs ALB
An honest ingress-nginx vs Traefik vs AWS ALB comparison from running all three in production: config model, performance, operational pain, and who should pick which.
Read the postClusterIP, NodePort, LoadBalancer: What You're Actually Buying
ClusterIP vs NodePort vs LoadBalancer explained by someone who's paid the bills: what each service type actually creates, how kube-proxy routes it, and when to use which.
Read the postDNS Works on My Laptop: CoreDNS Failures in Kubernetes
In-cluster DNS failing while your laptop resolves fine? Debug ndots:5 query amplification, the UDP conntrack race, CoreDNS health, and scaling, step by step.
Read the postImagePullBackOff: Every Cause I've Actually Hit
ImagePullBackOff has four real causes: bad tags, registry auth, rate limits, and network. Learn to read the pull error and fix the right one.
Read the postNode NotReady at 2AM: A Recovery Walkthrough
A node went NotReady at 2AM. Triage kubelet, disk pressure, and network partitions, then decide when to cordon, drain, or wait.
Read the postOOMKilled: Debugging Memory Limits Without Guessing
OOMKilled and exit code 137 mean the kernel killed your container. Learn to read Last State, separate cgroup OOM kills from eviction, and find the actual leak.
Read the postYour Pod Is Stuck in Pending: The Actual Checklist
Pod stuck in Pending? Run the real checklist: kubectl describe events, 0/N nodes available reasons, taints and tolerations, resource requests, and unbound PVCs.
Read the postForbidden: Debugging RBAC Without Handing Out cluster-admin
RBAC Forbidden errors decode completely if you read them. Use kubectl auth can-i, impersonation, and bindings to grant exactly what's missing.
Read the postAKS in Production: CNI Overlay, Workload Identity, Hidden Limits
CNI Overlay vs kubenet, Workload Identity, the invisible control plane, surge math, and the AKS quotas that only appear under load — from production.
Read the postArgoCD ApplicationSets: Stop Copy-Pasting Your Environments
Hand-maintaining near-identical ArgoCD Application manifests per cluster is a slow-motion incident. Here's how ApplicationSet generators fix it, and where they bite back.
Read the postArgoCD Sharding, Reconciliation Tuning, and the 1,000-App Wall
What actually breaks when ArgoCD grows past hundreds of Applications: controller sharding, reconciliation tuning, repo-server memory, and the metrics to watch.
Read the postArgoCD Sync Waves: Ordering Your Deploys Without Losing Your Mind
A practical guide to ArgoCD sync waves: why resource ordering matters, how the sync-wave annotation really works, and the mistakes that bite everyone.
Read the postCrashLoopBackOff: A Field Guide From Someone Who's Been There
CrashLoopBackOff isn't one error, it's five. A practical debugging guide to restart counts, last state, logs --previous, and the causes I actually see in production.
Read the postEKS vs AKS vs GKE at Scale: The Differences That Show Up in Production
An opinionated EKS vs AKS vs GKE comparison for teams running 10+ production clusters: pricing, forced upgrades, quota friction, and multi-cluster tooling.
Read the postEKS Networking: VPC CNI, Prefix Delegation, and the IP Math
ENI slot math, /28 prefix delegation, warm pool tuning, and the failure modes that page you at 2 AM — an operator's guide to EKS pod networking.
Read the postGitOps Is a Discipline, Not a Tool
Installing ArgoCD doesn't make you a GitOps shop. Here's why the tool changes nothing until the team agrees to stop running kubectl apply from their laptops.
Read the postGKE Autopilot Internals: What Google Actually Runs for You
How GKE Autopilot actually works under the abstraction: per-pod billing vs node provisioning, compute classes, upgrade mechanics, and where it quietly costs 2x.
Read the postkubectl apply Is Lying to You (and What to Do About It)
kubectl apply says 'configured' and you believe it. Here's why declared YAML and real cluster state drift apart, and how to actually close the loop.
Read the postInside the Kubernetes Scheduler: Why Your Pods Land Where They Do
A deep dive into kube-scheduler internals: filtering vs scoring, topology spread math, PriorityClass preemption, scheduler profiles, and why the descheduler exists.
Read the postThe Day Our Staging Cluster Ran Out of IP Addresses
Pods stuck in ContainerCreating, cryptic CNI errors, and a /24 subnet with zero free IPs. A war story about why 'just staging' still needs capacity planning.
Read the post