Mastering Progressive Delivery: Harness, Kubernetes, and Canary Phases

Your readiness probe passed. Every pod is green. The rollout finished in under two minutes. And three hours later, your Go API is silently returning malformed responses to roughly a quarter of your users. That is the scenario that keeps platform engineers up at night, and it is the exact scenario that a standard Kubernetes rolling update cannot prevent on its own.

So what actually happened there? A rolling update is a replacement mechanism. It scales down old ReplicaSets and scales up new ones, governed by two knobs: `maxUnavailable`, which limits how many pods can be down at once, and `maxSurge`, which limits how many extra pods can exist above your desired count. Kubernetes defaults both to twenty-five percent. The update finishes when the new pods pass their readiness probe, and that is, that is the critical gap. A readiness probe answers one question: is this container ready to accept traffic? It does not answer whether the new version of your API is actually behaving correctly under real production load. Think of it like a restaurant health inspection that checks whether the kitchen is open, not whether the food tastes right. Passing the inspection gets you customers. What happens after that is a different story entirely.

This is where the Google SRE team's concept of canarying becomes essential. Their definition is precise and worth sitting with: a canary is a partial, time-limited deployment of a change, followed by an evaluation step that decides whether to continue or stop. Evaluation is the operative word. You are not just deploying fewer replicas and hoping for the best. You are gathering evidence. Harness takes this model and structures it into two explicit groups inside a Kubernetes deployment stage. The first group is the Canary group. Harness creates a separate, temporary Deployment object for your new version, running a small number of pods alongside your existing stable workload. For your Go API, think of it as running, say, two pods of `api-v2` next to your ten pods of `api-v1`. Both versions sit behind the same Kubernetes Service, because the Service selector matches pods from both Deployments. That means a fraction of real traffic, proportional to the pod count, naturally reaches the canary. No special routing infrastructure required at this stage. The canary pods are real, they are serving real requests, and you can watch them. What gets me is how elegant that is — you are using the most basic Kubernetes primitive, a label selector, to split traffic without touching a load balancer config.

Now, the Canary group in Harness is explicitly temporary. After your verification steps run — health checks, log analysis, whatever you wire in — Harness executes a Canary Delete step, which removes that temporary Deployment entirely. Then, and only then, the Primary group begins. The Primary group is your main rolling update, using the replica count defined in your Service Definition manifests. The canary was never meant to stay; it was a controlled experiment with a hard end date. This two-phase structure is what separates Harness's approach from a naive "deploy fewer pods" strategy. Compare it to blue-green for a moment: blue-green runs two full environments simultaneously and flips all traffic at once, which means you need double the resources and your rollback is a traffic switch, not a pod deletion. A rolling update gives you no separate evaluation window at all. The Canary-Primary pattern sits between those two: limited resource overhead, a real evaluation gate, and a clean promotion path. The trade-off, and there is always a trade-off, is that you are running two versions of your API simultaneously during the canary window. For a Go service, that means you need to think carefully about database schema compatibility, gRPC contract versions, and any shared state. A migration that drops a column your old pods still read will cause failures the canary window cannot catch, because the old pods are still running.

Marco, you already know Kubernetes well enough to feel the gap between "the rollout succeeded" and "the release is healthy." That intuition is exactly right, and it is the foundation this entire pattern is built on. A green rollout is not evidence of a good release. It is evidence that your pods started. The Canary-Primary model forces you to treat deployment as a decision process, not a fire-and-forget command. You deploy a small slice, you observe it under real conditions, and you make a deliberate choice to promote or revert. That decision gate is the blast radius control. If your canary pods start returning five-hundred errors, you delete two pods, not ten. Your users barely notice. Without that gate, you are betting your entire production fleet on the same readiness probe that passed in staging. And Marco, that bet is one most senior engineers have lost at least once. The architecture we are going to build together is specifically designed so you never have to lose it again.

The alert fired at two forty-seven in the morning. The canary had already been deployed, isolated, and given its configured traffic share—the controlled topology from the previous lesson. Every pod was Running. Every readiness probe was returning two hundred. Harness had promoted the canary to Primary forty minutes earlier, and the deployment log showed nothing but green checkmarks. And yet, roughly one in five requests to the Go API was timing out. Not failing with a five-hundred. Timing out. Silently. The kind of failure that takes twenty minutes to surface in dashboards and another twenty to believe. The routing experiment worked as configured. The failure was in the promotion decision: the evidence used to declare the new version safe was insufficient.

Now, the belief that readiness probes are sufficient is not naive. It has a real, defensible version. A readiness probe, after all, is a live signal from inside the container. It runs throughout the pod's entire lifecycle, not just at startup. If the pod becomes overloaded, a well-designed readiness check can withdraw it from Service endpoints automatically, protecting users. That is genuinely powerful. And a startup probe can shield slow-starting applications, holding liveness and readiness checks back until the process is actually ready to be evaluated. So the argument goes: if you write your probes carefully, Kubernetes handles the rest.

That argument holds until you test it against what a readiness probe actually measures. A readiness check answers one question: should this container receive traffic right now? It does not ask whether your API routes are returning correct data. It does not ask whether your downstream database is responding within acceptable latency. It does not ask whether the, the new version introduced a regression in your payment processing logic. It is like a smoke detector tuned to a particular condition. The detector is working perfectly. The food on the stove has been burning for ten minutes. A successful readiness probe is evidence for the specific condition the check tests, and nothing more.

There is a second failure mode that compounds this, and it is, it's one I find genuinely alarming. Liveness and readiness probes have different consequences, and mixing them up causes real damage. A failed readiness probe removes the pod from Service endpoints. A failed liveness probe tells the kubelet to kill and restart the container. Google's SRE guidance documents a feedback loop where health-check-based restarts under load can worsen the overload, because each restart adds work to the remaining pods. Restarting a struggling pod can worsen the problem. Sometimes it is acceleration toward collapse. So making your readiness probe more comprehensive, more aggressive, is not automatically safer. A probe that performs expensive work adds load precisely when the service is already degraded.

So, what does a real canary gate look like? Harness first confirms that canary pods reach steady state: Running and passing readiness. Then its verification gate must go further. For your Go API, Marco, compare canary error rate, latency, and resource saturation against the Primary workload's baseline, rather than relying on absolute thresholds alone. That comparison is the, the critical piece. A canary returning two percent errors sounds alarming until you see the Primary is also at two percent. Conversely, a canary at one percent errors is a problem if Primary is at zero point one. The number alone tells you almost nothing. The delta tells you everything.

And this is where low canary traffic becomes its own hazard, Marco. Suppose your canary receives fifty requests during the verification window. A single slow response can skew a latency percentile dramatically. That is not a signal. That is noise wearing a signal's clothes. Make minimum request count a Harness promotion gate, not merely a time window. A five-minute window with three hundred requests is evidence; a five-minute window with forty requests is a guess. Define the inconclusive path in advance: hold the canary and extend collection when safe, or roll back when telemetry is unavailable, the window cannot be extended, or a safety limit is breached. Inconclusive is not healthy, and it is not an automatic promotion.

A green readiness status should not override failing user-facing metrics. That is the promotion rule. Define it before deployment: promote only when steady state passes, the minimum sample is met, and canary metrics remain within the agreed error-rate, latency, and saturation deltas from Primary. Roll back when a guardrail fails; hold when evidence is insufficient. Readiness, liveness, and startup probes protect different parts of the container lifecycle. Harness verification decides whether the release is safe for users. The routing mechanics got us a controlled experiment. These gates decide whether Harness promotes that experiment to Primary or stops it.

Once the verification gate from the previous lecture approves promotion, Harness enters the Primary phase. Now focus on the mechanics: what happens next, in what order do workloads and traffic move, and what breaks if that order is wrong? That operational sequence is the part many pipeline descriptions reduce to a single bullet point.

So, the canary has passed its gates. What Harness does next is not a simple label swap. The Primary Deployment group performs a Kubernetes rolling update of the actual Deployment object, targeting the replica count defined in your Deployment manifest. It is like a relay race: the canary tests the route, but the baton now passes to the Primary, which runs the full distance. The rolling update is governed by `maxSurge` and `maxUnavailable`, the same knobs you already know from standard Kubernetes rollouts. The Primary rollout must reach steady state before anything else is decommissioned. That sequencing is not optional.

Now, this is the part the polite version of this conversation usually skips. Deleting the canary workload before the new Primary pods are fully Ready creates a gap. During that gap, your routing layer may have no healthy backend for the new version, and requests can fail silently. In the newer Kubernetes Canary flow, canary capacity is typically scaled down as traffic shifts to Primary rather than being deleted before the rollout. Depending on the routing integration, weights may move from ten percent to twenty-five, fifty, and finally one hundred percent. The canary pods are reduced in parallel with those weight changes, not before them. Complete cleanup only after the Primary Service has one hundred percent of traffic and the new Primary pods are Ready.

Suppose you tried a shortcut: instead of running the Primary rolling update, you just relabeled the canary pods as stable. I have seen this proposed more than once. The problem is that canary pods were provisioned at canary scale, typically two or three replicas. Your production Deployment manifest specifies ten, or twenty, or whatever your capacity model requires. Relabeling gives you the right image on the wrong number of pods. Beyond replica count, the Primary rolling update also ensures that Kubernetes records a new rollout revision, which means `kubectl rollout undo` can revert to the previous stable image if something surfaces after promotion. Relabeling leaves no revision trail. That means your rollback story disappears.

Marco, here is where your Go API needs specific attention. During the rolling update, your Go API needs to handle termination signals gracefully. Your Go HTTP server must catch that signal and stop accepting new connections while finishing in-flight requests. The `terminationGracePeriodSeconds` field should be set longer than your API's maximum expected request duration plus any connection-draining time. If a background job or a message-consumer offset commit is in progress, that window needs to cover those too. A server that exits immediately on SIGTERM will drop keep-alive connections mid-stream, and those failures will appear in your canary metrics window as noise, or worse, as false positives that survive into the Primary rollout. The, the signal handling is not a deployment concern. It is a correctness concern that the deployment surface makes visible.

Configure a timeout for the Primary rollout and for each traffic-shift operation. If a Deployment stalls because pods cannot become Ready, an image cannot be pulled, or capacity is unavailable, Harness must fail within a known interval rather than wait indefinitely. Investigate the Deployment conditions, Events, ReplicaSets, and Service endpoints; then either correct the issue and retry or roll back with the retained Deployment revision. Do not leave a partially promoted workload consuming production traffic without an explicit operational decision. After completion, confirm that the Primary Service endpoints match the desired replica count, the rollout is complete, traffic is stable at one hundred percent, and the prior revision remains available for rollback.