Kubernetes 1.37 Upgrade Guide: What Platform Teams Should Test First

Kubernetes 1.37, code-named Garhwal, is not an upgrade to approach by enabling every new feature gate. The safer strategy is to identify behavior that changes by default, test the workloads most exposed to it, and keep alpha features outside the critical path.

This guide turns the release notes into an operator-focused upgrade plan. It does not replace the version-skew policy or your managed Kubernetes provider's release notes.

What Matters Most in Kubernetes 1.37

The release contains changes across API machinery, scheduling, storage, observability, and node operations. Four areas deserve early attention.

First, ConcurrentWatchObjectDecode is enabled by default. Kubernetes reports that concurrent decoding reduced watch-cache initialization time by about 40% in a benchmark with more than 150,000 Pods, and by about 55% when combined with EtcdRangeStream. The trade-off is burstier conversion-webhook traffic because several conversions may execute concurrently.

Second, Memory QoS is beta and enabled by default on supported Linux nodes using cgroup v2. That makes memory behavior a workload test concern, especially for applications that run close to their limits.

Third, native histograms for Kubernetes metrics are beta and enabled by default. Before changing dashboards or recording rules, verify that every component in the metrics path can ingest and query them correctly.

Fourth, the release adds new capabilities such as Node Lifecycle Conditions and workload-aware scheduling. Some are alpha, disabled by default, or provide shared API vocabulary without changing core controller behavior. Treat those as separate adoption projects, not reasons to modify the base upgrade.

Build a Risk Inventory Before the Upgrade

Start with the components that extend Kubernetes behavior:

kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
kubectl get crd
kubectl get apiservices
kubectl get --raw /metrics > apiserver-metrics-before.txt

Record the versions of the CNI, CSI drivers, ingress controllers, service mesh, policy engine, metrics stack, backup tooling, and operators. For a managed cluster, check the provider's supported add-on matrix rather than assuming upstream compatibility guarantees apply unchanged.

Pay special attention to conversion webhooks. A CRD with multiple stored or served versions may call a conversion service while the API server initializes caches. Confirm that the webhook has enough replicas, sensible timeouts, and metrics for latency and failures.

Capture a baseline before upgrading:

• API request latency and error rate
• conversion-webhook latency, concurrency, and failure rate
• scheduler pending-Pod latency
• kubelet and container runtime errors
• node memory pressure and OOM events
• controller workqueue depth
• Prometheus ingestion errors and query correctness

Without a baseline, an upgrade can be technically successful while quietly making reconciliation slower.

Upgrade a Representative Non-Production Cluster

Use production-like CRD counts, webhook behavior, node shapes, and workloads. An empty test cluster will not expose the problems most likely to appear under load.

Run API deprecation checks against manifests and live objects. Then upgrade the control plane before nodes, following the provider's supported sequence and Kubernetes version-skew rules. Keep the first node pool small enough to isolate but large enough to host representative workloads.

After the control-plane upgrade, test these paths before expanding the rollout:

1. Create, update, watch, and delete a high-volume custom resource.
2. Restart an operator and observe cache initialization.
3. Roll a workload with strict memory requests and limits.
4. Drain and replace a canary node.
5. Exercise HPA behavior and inspect relevant controller metrics.
6. Run backup and restore smoke tests for stateful applications.
7. Validate dashboards, alerts, and long-running Prometheus queries.

Do not interpret a healthy /readyz response as complete validation. It proves that the control plane is ready, not that every extension and workload remains correct.

Keep Alpha Features Out of the Base Change

Kubernetes 1.37 introduces useful alpha work, including Node Lifecycle Conditions and scheduler improvements for in-place Pod resize. Alpha APIs and feature gates may change and are commonly disabled by default.

Upgrade first with the provider-supported defaults. If an alpha feature solves a real problem, test it later in a separate environment with its own success criteria and rollback plan. Combining a version upgrade with experimental scheduling, node, or storage behavior makes failures harder to attribute.

This separation is especially important for Node Lifecycle Conditions. In 1.37 the named conditions are a status channel. Core workload controllers do not automatically cordon, drain, or reschedule Pods because a condition appears.

Roll Nodes in Controlled Waves

For node upgrades, use a small canary pool and verify:

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get events -A --sort-by=.lastTimestamp

Check PodDisruptionBudgets before draining. A stalled drain is usually a workload availability problem that automation has correctly surfaced, not a reason to force eviction immediately. Our Kubernetes PDB blocking node drain guide covers that failure mode.

Expand in waves only after the canary pool completes normal traffic, autoscaling, restarts, and a deliberate drain. Pause when error budgets, scheduling latency, webhook failures, or memory-pressure signals move beyond the agreed threshold.

Define the Rollback Boundary

A Kubernetes control-plane downgrade is often unsupported, especially on managed services. The practical rollback unit is therefore usually the node pool, add-on, workload release, or feature gate—not the control-plane version.

Before starting, document which actions are reversible:

• retain or recreate an older node pool if the provider supports it;
• pin compatible add-on versions;
• roll back application manifests independently;
• disable optional feature gates introduced during testing;
• restore data only through a tested backup process.

Do not promise a cluster downgrade unless your provider explicitly supports it.

Upgrade Decision

Kubernetes 1.37 offers meaningful performance and operational improvements, but the right rollout is deliberately boring: inventory extensions, measure a baseline, test conversion webhooks and memory-sensitive workloads, canary the nodes, and keep experimental features separate.

The release headline is not the change plan. Your dependency graph is.

Internal Links

• Build a Kubernetes upgrade impact analyzer
• Fix nodes that stay NotReady after an upgrade
• Understand Kubernetes nodes

Sources

• Kubernetes: v1.37 Garhwal release
• Kubernetes: version skew policy
• Kubernetes: Memory QoS graduates to beta
• Kubernetes: native histograms graduate to beta