Hi, I'm Khwelisile Lukhuleni a Senior Platform Engineer

The mega-module trap

The first version of our platform modules did too much. One eks-platform module created the cluster, node groups, IAM roles, the VPC endpoints, and half the observability stack. It felt productive — one module block and you had a platform. Then the second team needed a slightly different node group topology, and the third team needed the cluster without the VPC endpoints, and every "small" option became another variable. Within six months the module had 70+ inputs and nobody could predict what a change would do.

The fix was boring and it worked: flat modules, one concern each, composed at the environment level. The networking module doesn't know compute exists. The cluster module takes subnet IDs as inputs; it never looks them up itself. Composition happens in a thin per-environment root that reads like a manifest:

module "network"  { source = "…//vpc"        version = "2.4.1" … }
module "cluster"  { source = "…//eks"        version = "3.1.0"
  subnet_ids = module.network.private_subnets }
module "o11y"     { source = "…//monitoring" version = "1.9.2" … }

State is a blast radius, not a database

Our second hard lesson: one state file per environment sounds tidy until terraform plan takes eleven minutes and two teams hit lock contention every afternoon. We split state along service boundaries within each environment — network, cluster, shared services, then one workspace per team's infrastructure. Plans dropped back under a minute, and more importantly, the blast radius of a bad apply shrank to the boundary of one state file.

Version pins are a feature, not paranoia

Every module reference is pinned to an exact version. Upgrades are pull requests that roll dev → qa → prod, with the plan output posted into the PR by CI. Two reviewers: someone from the consuming team, someone from platform. The team validates intent; platform validates that the change won't create a cascading failure. It sounds heavyweight — in practice it takes minutes, because the modules are small enough that diffs are readable.

What I'd tell you to steal

• One concern per module. If the name needs an "and", split it.
• Compose at the root, pass IDs explicitly — no data-source archaeology inside modules.
• Split state by environment and service boundary before plans get slow, not after.
• Exact version pins everywhere, upgrades as reviewed PRs with plan output attached.
• Ship a module template repo — the fastest way to keep standards is to make the right way the lazy way.

The problem with workers

Classic Airflow puts your task code and the scheduler in the same Python environment. The first time a data team needs pandas 2.x while another DAG pins 1.5, you're in dependency hell — and upgrading Airflow itself becomes a cross-team negotiation. We moved every task to KubernetesPodOperator: each task is a container image the owning team builds, and Airflow's environment contains Airflow and nothing else.

That one decision decoupled three lifecycles that had been tangled together: the scheduler's upgrade cadence, each pipeline's dependency set, and the platform's node capacity. Teams ship task images through the same CI as their services. Airflow upgrades stopped being scary.

Self-healing is configuration, not magic

The reliability wins came from three settings applied ruthlessly:

• Retries with backoff on every task. Transient EKS scheduling hiccups and spot interruptions stopped paging anyone — they retry and pass.
• Execution timeouts on every task. A hung pod is worse than a failed one; timeouts turn "silently stuck since 3am" into a clean failure with a page.
• Pools around shared backends. One team's 400-task backfill used to brown-out the warehouse for everyone. Pools cap concurrent connections per resource, so a backfill is a queue, not an incident.

Pay for what runs, not what waits

With per-task resource requests, heavy transforms schedule onto a dedicated node group with cluster-autoscaler behind it. When the DAG finishes, the nodes drain and disappear. Our old static worker fleet was sized for the nightly peak and idle 80% of the day — the pod model cut that waste out entirely, and Cost Explorer finally attributed spend to the team whose image was running.

The observability contract

Every DAG exports success/duration metrics to Prometheus, and each pipeline has an SLO. The routing rule that made this sustainable: a missed SLO pages the pipeline's owning team, not the platform on-call — with the run ID, task logs and the Grafana panel in the alert. The platform team gets paged when the scheduler or the cluster is unhealthy. Ownership boundaries in the alerting config are what keep a shared Airflow from becoming a shared burden.

CPU alerts are a lie detector that always lies

The alert estate I inherited was classic: CPU over 80%, memory over 90%, disk over 85% — per node, per service, per environment. Pager volume was high, and none of it answered the only question that matters: are users having a bad time right now? A service can run hot and be perfectly healthy; it can idle at 20% CPU while every request times out on a dead dependency. Resource alerts measure the machine's comfort, not the user's experience.

So the platform's rule became: page on symptoms, graph the causes. The only thing that wakes a human is a service-level indicator moving toward breaching its objective. CPU, memory and saturation still exist — as dashboard context for the person already awake.

Two SLIs cover almost everything

For each service we define exactly two SLIs at the edge that matters (the ingress, the topic, the DAG): availability — the fraction of requests that succeed — and latency — the fraction served faster than the threshold users notice. Each gets an objective agreed with the owning team, and the gap between objective and reality is the error budget. That's it. Teams that wanted ten bespoke SLIs got talked down to two; the discipline is the feature.

Burn rate, not thresholds

A static "error rate > 1%" alert has two failure modes: it pages at 3am for a blip that self-heals, and it stays silent while a slow leak quietly spends a month of budget. Multi-window burn-rate alerts fix both:

# page: burning 5% of monthly budget in an hour
(burn_rate_5m > 14.4 and burn_rate_1h > 14.4)
# ticket: slow leak, look tomorrow
(burn_rate_6h > 1 and burn_rate_3d > 1)

Fast burn pages a human immediately; slow burn opens a ticket. The 5m/1h pairing stops flapping — both windows must agree before anyone's phone makes a sound.

The routing rule that made it stick

Every ServiceMonitor carries a team label, and Alertmanager routes on it: an SLO burn pages the service's owning team; the platform team is paged only when the platform itself is the problem — cluster, ingress, Prometheus, the pipeline. The alert body carries the Grafana panel, the runbook link and the current budget remaining, because an alert without context is just anxiety with a timestamp.

Dashboards as code, or they rot

Every service gets one provisioned golden dashboard — SLI trend, budget remaining, saturation, dependencies — generated from the same Git repo that defines the SLOs. Hand-edited dashboards drift, multiply and lie; provisioned ones stay honest because a PR is the only way to change them.

What actually changed

• Pager volume dropped hard once symptom-based paging replaced threshold noise — most old alerts became dashboard panels or got deleted.
• Time-to-fix dropped because the page arrives with the graph, the runbook and the budget math already attached.
• Error budgets turned reliability arguments into arithmetic: burn the budget, spend the sprint on stability; keep it, ship features. Nobody has to win a meeting.

Summary

A routine Helm release bumped a shared ingress controller. The new version changed a default: connection draining dropped from 300s to 30s. Rollout health checks passed — new pods were Ready, endpoints healthy. But long-lived internal gRPC connections were severed mid-stream across every namespace behind that ingress. Downstream services saw connection resets, retried aggressively, and the retry storm pushed p99 latency past timeout budgets. Impact: ~40 minutes of degraded internal APIs. No customer data affected.

Timeline (condensed)

• 14:02 — ArgoCD syncs the ingress chart bump. Rollout completes green.
• 14:04 — First alerts: gRPC error-rate SLO burn on two services. On-call assumes app issue.
• 14:11 — Three more teams paged. Pattern recognized: every affected service shares the ingress.
• 14:19 — Rollback initiated via ArgoCD history. Drain settings restored.
• 14:42 — Error rates back within SLO. Incident closed.

What actually failed

Not the ingress controller — it did exactly what its new defaults said. What failed was our model of "safe change": rollout checks validated that the new pods were healthy, but nothing validated the behavior of existing traffic during the transition. Readiness is a statement about the future; draining is a statement about the present. We were only testing one of them.

What changed afterwards

• Upgrade diffs for shared infrastructure now include a rendered-defaults diff — helm template old vs new, so changed defaults show up in review even when our values file doesn't change.
• Shared components roll through a canary ingress class first, with synthetic long-lived connections attached — the exact failure mode this incident would have caught.
• Alert routing got a "shared infra" correlation rule: three teams paging on the same dependency escalates to platform immediately, instead of three parallel investigations.
• Blameless review, as always. The engineer who shipped the bump also wrote the canary tooling — that's how you keep people shipping.

The lesson I keep re-learning

Green checks mean your change passed the tests you thought to write. The riskiest changes are the ones that alter behavior you never encoded as a test — and shared-infrastructure defaults are exactly where those live.