Skip to content

Release notes for kOps 1.37 series

⚠ kOps 1.37 has not been released yet! ⚠

This document collects release notes before the final kOps 1.37 release.

kOps 1.37 adds support for Kubernetes 1.37. It removes gossip DNS, AWS Classic Load Balancer support for the API, and Kopeio networking. It updates the default container runtime and etcd versions. It also expands support for Karpenter and Akamai (Linode), adds private asset repositories, and improves node bootstrap and IAM security.

Significant changes

  • kOps 1.37 removes AWS Classic Load Balancer (CLB) support for the API, which kOps deprecated in 1.26. The breaking changes section gives migration instructions.

  • kOps 1.37 removes Kopeio networking because the project is no longer maintained. The breaking changes section gives migration instructions.

  • On AWS, spec.cloudProvider.aws.useIPBasedNodeNames uses the EC2 private DNS name as the Kubernetes node name. For example, a node name can be ip-10-0-0-1.eu-west-1.compute.internal instead of an EC2 instance ID. In the v1alpha2 API, use spec.cloudConfig.useIPBasedNodeNames. This setting restores the naming that kOps used before 1.24. It affects only new nodes. A cluster roll is necessary to rename existing nodes.

  • containerd 2.3.4 and runc 1.4.3 are now the defaults. If you pin spec.containerd.version to 2.3, use version 2.3.4 or newer. Versions 2.3.0 through 2.3.3 cannot start pods when kOps pins the sandbox image by digest. (containerd#13529)

  • kOps no longer supports containerd versions older than 2.1.0. The breaking changes section gives migration instructions.

  • Kubernetes 1.37 clusters now use etcd 3.7.0 by default. For older supported Kubernetes versions, kOps uses etcd 3.6.12 or 3.5.31. (#18557)

Removal of gossip DNS

kOps 1.29 deprecated gossip DNS, and kOps 1.37 removes it. kOps rejects new gossip clusters. Existing gossip clusters must migrate before an upgrade to kOps 1.37. To check a .k8s.local cluster, inspect spec.networking.topology.dns.type in the cluster spec. The cluster uses gossip and is affected unless this field is set to None.

Use kOps 1.36 to migrate to --dns=none or a hosted DNS zone before the upgrade. Hybrid bootstrap in kOps 1.36 makes this migration easier. The gossip documentation gives the migration steps.

Gossip DNS was the only remaining purpose of protokube. kOps no longer builds, ships, or runs protokube. The gossipConfig and dnsControllerGossipConfig fields are deprecated and have no effect. Remove these fields from cluster specs.

Other changes of note

Cluster lifecycle and tooling

  • The new kops validate cluster --max-unready-nodes flag lets validation tolerate a specified number of non-ready worker nodes. (#18614)

  • During cluster creation, kOps now reports an error for empty and sign-only (+ or -) feature-gate entries. Previously, these entries caused a panic or an empty key. (#18408)

  • kOps now preserves AWS and GCE binariesLocation settings when it stores cluster specs through the v1alpha2 API. Previously, kOps removed these settings without a warning. (#18740)

Node bootstrap and container runtime

  • Nodes in IPv6-only clusters that use --dns=none now bootstrap correctly. kOps omits unreachable IPv4 control-plane endpoints. It also tries each available configuration server instead of using the full retry period on the first server. (#18684)

  • Nodes now download an xz-compressed nodeup binary, which is approximately one quarter of the initial size. Custom node images must provide xz. All supported distributions already provide it. (#18737)

  • The bootstrap script now requires curl to download nodeup. All supported node images provide curl. Custom images must also provide it.

  • Node bootstrap now limits the kops-controller retry delay and prevents client-side throttling of bootstrap node lookups. It also returns task errors instead of exiting. These changes help nodes recover when the control plane starts slowly or many nodes join at the same time. (#18675, #18718, #18742)

  • On RHEL 10-family operating systems, nodeup installs kernel-modules-extra. This package supplies the kernel modules that networking components require. (#18724)

  • kOps now recognizes Amazon Linux 2027, which is in public preview. Support is experimental: AWS does not recommend the preview for production workloads and will deprecate the preview AMIs at general availability. Amazon Linux 2027 differs substantially from Amazon Linux 2023. It is based on Fedora 44 and 45, and it ships kernel 7.1, systemd 260, glibc 2.44, and DNF 5. It also enables SELinux in enforcing mode by default and supports cgroup v2 only. (#18777)

  • kOps now sets containerd configuration version 4 when it installs containerd 2.3 or newer. containerd versions 2.3.0 through 2.3.4 do not start if a drop-in configuration has a higher version than the root configuration. The NVIDIA container toolkit can write such a drop-in configuration. Flatcar and ContainerOS continue to use version 3 because they use the operating system's containerd package. If spec.containerd.packages overrides the package URL, spec.containerd.version must match the packaged binary. kOps now also uses this field to select the configuration version. (#18728)

  • With containerd configuration version 4, containerd ignores top-level grpc, ttrpc, metrics, and debug addresses in spec.containerd.configAdditions. kOps no longer migrates entries from the legacy plugins."io.containerd.grpc.v1.cri" plugin. Use the plugins."io.containerd.cri.v1.runtime" and plugins."io.containerd.cri.v1.images" paths instead. A spec.containerd.configOverride for containerd 2.3 or newer must declare version = 4.

  • Release binaries are much smaller. kOps removed unused cloud and Kubernetes clients from nodeup and enabled linker method pruning. It also removed dependency-heavy runtime paths that were not used. (#18552, #18554, #18571, #18572, #18687, #18695, #18710, #18721)

AWS

  • Karpenter support now includes these changes:
  • Generated EC2NodeClass objects now map spec.rootVolume to blockDeviceMappings. Thus, Karpenter-created instances use the configured root volume. (#18655)
  • spec.mixedInstancesPolicy.instanceRequirements now creates CPU, memory, and excluded-instance requirements in the NodePool. When instance requirements are set, spec.machineType and spec.mixedInstancesPolicy.instances are optional. kOps rejects invalid Karpenter configurations before it renders addons. (#18676, #18761)

  • Karpenter's iam:PassRole permission now applies only to EC2 and the worker node role. With a custom instance profile, Karpenter can pass any role to EC2. Mutating EC2 actions now require the cluster and karpenter.sh/nodepool tags. (#18582)

  • For an AWS subnet specified by ID, the cluster spec can omit the zone. kOps gets the zone from EC2. (#18630)

  • kOps now rejects V2 bootstrap tokens when their pre-signed STS URLs do not use HTTPS. (#18574)

  • AWS VPC CNI node roles now include ec2:DescribeSecurityGroups. The CNI requires this permission for subnet discovery. Without it, the CNI allocates addresses only from the primary ENI's subnet. (#18693)

  • On AWS, kOps no longer grants Route 53 permissions to the control-plane role when spec.iam.useServiceAccountExternalPermissions is enabled. Grant these permissions through spec.additionalPolicies or a service account IAM role.

  • On AWS, kOps limits the ec2:ModifyInstanceAttribute permission for kube-router and kindnet to instances that have the cluster tag.

  • On AWS, kOps grants ec2:DescribeInstanceTypes only to instance roles that require it. Nvidia GPU support and Cilium ENI IPAM require this permission. AWS VPC CNI clusters retain it as a CNI permission. Cluster autoscaler permissions omit it when spec.clusterAutoscaler.awsUseStaticInstanceList is enabled.

  • On AWS, kOps no longer grants ec2:DescribeRegions to etcd-manager. etcd-manager now reads the region from the instance metadata service.

  • On AWS, KMS policies for customer managed keys restrict kms:CreateGrant to grants that AWS services create for the role. The policies restrict the other KMS actions to calls through EC2 or S3. When spec.encryptionConfig is enabled, control-plane roles keep unrestricted data actions so that a KMS plugin can call KMS directly. Other direct KMS calls require spec.additionalPolicies or a service account IAM role.

  • On AWS, kOps no longer creates inline IAM policies for roles that require no permissions. This includes the bastion role and the default worker role. kops update cluster deletes earlier inline policies from these roles. The Terraform target removes the related aws_iam_role_policy resources during the next apply.

Azure

  • Azure nodes now prove their identity to kops-controller with a PKCS7-signed attested metadata document. kOps checks the signature chain, nonce, expiration, and signed VM ID before it accepts the bootstrap request. (#18163)

GCP, DigitalOcean, Hetzner, and Scaleway

  • On DigitalOcean, Hetzner, and Scaleway, control-plane user data no longer contains cloud API credentials. kOps supplies credentials to applicable components through generated manifests. Scripts in additionalUserData and units in spec.hooks must now set the credentials that they use.

  • During GCE cluster deletion, kOps no longer selects unrelated empty backend services. It also removes routes and load-balancer firewall rules that can block network deletion. (#18460, #18760)

  • When kOps deletes a DigitalOcean cluster, it now removes the SSH key that it uploaded for the cluster. (#18697)

Cloud controller managers and CSI drivers

  • kOps updates the default versions of the cloud controller managers and CSI drivers that it deploys:
Provider Component Version
AWS cloud-controller-manager v1.37.0
AWS EBS CSI driver v1.65.0
AWS ecr-credential-provider v1.37.0
GCP cloud-controller-manager v36.2.4
GCP PD CSI driver v1.26.0
Azure cloud-controller-manager and cloud-node-manager v1.36.5
Azure azuredisk CSI driver v1.34.5
DigitalOcean cloud-controller-manager v0.1.69
DigitalOcean CSI driver v4.18.0
Hetzner cloud-controller-manager v1.36.0
Hetzner CSI driver v2.22.2
Akamai (Linode) cloud-controller-manager v0.9.8

kOps applies the addon changes during kops update cluster. The ecr-credential-provider binary is part of the node image, so nodes install it when they are replaced.

  • On DigitalOcean, kOps now sets a providerID on nodes. The cloud controller manager needs it from v0.1.58 on, because those releases resolve a node through the InstancesV2 interface. The field is immutable, so only nodes created after this change carry it. Nodes already in the cluster keep the providerID that v0.1.45 recorded for them, so both kinds work with the version that kOps now deploys.

  • On DigitalOcean, the cloud controller manager now creates a network load balancer for a new LoadBalancer Service that carries no service.beta.kubernetes.io/do-loadbalancer-type annotation. Earlier versions created a regional load balancer. A network load balancer passes traffic through and forwards TCP and UDP only. It does not support HTTP, HTTPS, or HTTP/2 forwarding rules, TLS termination, or the PROXY protocol. Existing load balancers keep their current type, because the controller records the type on the Service before it updates the load balancer. To keep the earlier behavior for a Service, set the annotation to REGIONAL.

  • On Hetzner, the cloud controller manager and CSI driver versions that kOps deploys support Kubernetes 1.33 and newer. Hetzner removed Kubernetes 1.32 from its support matrix in CSI driver v2.21.0. A Kubernetes 1.32 cluster on Hetzner runs untested component versions. Support for Kubernetes 1.32 is deprecated in kOps 1.37.

  • The AWS EBS CSI driver addon no longer creates the ebs-csi-controller-volumemodifier Service. kOps does not deploy the volume-modifier-for-k8s sidecar that this Service selects, because the sidecar image is published only to a registry that kOps does not mirror. The Service has had no endpoints since kOps 1.36, and volume modification through the ebs.csi.aws.com annotations has been unavailable for the same period. This addon does not prune, so an existing cluster keeps the Service until you delete it with kubectl delete service -n kube-system ebs-csi-controller-volumemodifier.

Private asset repositories

  • Private asset repositories now support GCE gs:// URLs for KOPS_BASE_URL and spec.assets.fileRepository. Nodes use their instance service account. The account must have roles/storage.objectViewer on the bucket. (#18466, #18623, #18713)

  • Private asset repositories now support AWS s3:// URLs for KOPS_BASE_URL and spec.assets.fileRepository. Nodes use their instance credentials. The asset repository documentation lists the required permissions. AWS nodeup downloads require curl 8.0 or newer. (#18661)

  • Private asset repositories now support Azure azureblob://<account>/<container>/<prefix> URLs for KOPS_BASE_URL and spec.assets.fileRepository. Nodes use their system-assigned managed identity. Grant the Storage Blob Data Reader role on the asset container to this identity. The asset repository documentation gives more information. (#18666)

Networking

  • kOps now uses Calico v3.31.6, AWS VPC CNI v1.23.0, and Flannel v0.28.9. The default CNI plugins version is v1.7.1 for Kubernetes 1.34 and v1.8.0 for Kubernetes 1.35. Kubernetes 1.36 or newer uses v1.9.1. Kubernetes 1.32 and 1.33 continue to use v1.6.2. (#18701, #18692, #18704, #18705)

  • kindnet now mounts the containerd NRI socket for faster local pod insertion. DNS caching is disabled by default. Set spec.networking.kindnet.dnsCaching: true to keep the previous behavior. (#18635, #18727)

  • The CoreDNS PodDisruptionBudget now uses maxUnavailable: 33% instead of 50%. With three or more replicas, voluntary disruptions leave at least two DNS pods running. The budget also sets unhealthyPodEvictionPolicy: AlwaysAllow. This setting lets a node drain evict CoreDNS pods that are running but not ready.

  • kOps now uses kube-router v2.11.1. The new spec.networking.kubeRouter.useNFTablesForNetpol field uses nftables instead of iptables and ipsets for network policy enforcement. The nftables backend is experimental. This field does not change the service proxy or router controllers.

  • The new externalIPRanges and loadBalancerIPRanges fields restrict the CIDRs that kube-router accepts for Service external and load-balancer IPs. Set these fields in spec.networking.kubeRouter. Either field enables strict external IP validation. kube-router does not program or advertise addresses outside the specified ranges. If the fields are not set, kube-router continues to accept all addresses.

  • The kube-router DaemonSet now uses a startup probe before its liveness probe. Thus, a slow informer sync no longer causes a startup CrashLoopBackOff. The DaemonSet also sets hostIPC: true, which kube-router requires for direct server return (DSR).

Etcd

  • etcd-manager now mounts the bundled etcd images as image volumes on all supported Kubernetes versions. For versions older than Kubernetes 1.36, kOps enables the ImageVolume kubelet feature gate on control-plane nodes. This support now includes Kubernetes 1.32. kOps no longer builds or publishes the kops-utils-cp image that previously copied etcd binaries.

Akamai (Linode) (experimental)

  • The provider can now manage SSH keys, networks, subnets, instances, and block-storage volumes. This support includes etcd data volumes. The provider also supplies a default network CIDR and kops-controller configuration. (#18383, #18590, #18622, #18700, #18709, #18743, #18744)

  • User-facing references now use "Akamai (Linode)" instead of "Linode". The API remains unchanged. (#18691)

Breaking changes

  • kOps no longer supports an AWS Classic Load Balancer (CLB) for the API. A cluster fails validation if spec.api.loadBalancer.class is Classic or is not set, which previously selected Classic. kOps also removes the deprecated kops create cluster --api-loadbalancer-class flag. Existing clusters must use kOps 1.36 or earlier to migrate to a Network Load Balancer (NLB). See the CLB to NLB migration guide. This change does not affect externally managed CLBs that use spec.externalLoadBalancers[].loadBalancerName.
  • kOps no longer supports gossip DNS. Existing gossip clusters must use kOps 1.36 to migrate to --dns=none or a hosted DNS zone.
  • kOps releases no longer include the protokube binary. Nodes no longer install or run it.
  • The spec.gossipConfig and spec.dnsControllerGossipConfig fields are deprecated and have no effect. Remove them from cluster specs.
  • kops toolbox template no longer resolves template expressions to methods on rendered values. This restriction also applies to values from template functions. Rewrite expressions such as {{ now.Unix }} to use struct fields, map keys, or registered template functions.

Other breaking changes

  • kOps no longer supports Kopeio networking because the project is not maintained. Clusters with spec.networking.kopeio now fail validation. kops create cluster no longer accepts --networking kopeio or --networking kopeio-vxlan. With kOps 1.36 or earlier, switch to another CNI and roll the cluster. The kindnet CNI provides similar minimal networking. Complete this migration before an upgrade to kOps 1.37.

  • kOps no longer supports containerd versions older than 2.1.0. A spec.containerd.version value below 2.1.0 fails validation. The 1.6, 1.7, and 2.1 release lines have reached end of life, and version 2.0 reaches end of life in March 2027. Remove the pinned version to use the default version, 2.3.4. You can also pin version 2.1.0 or newer. Nodes install the new runtime when they are replaced.

Known Issues

  • None at this time

Deprecations

  • Support for Kubernetes version 1.31 is removed in kOps 1.37.

  • Support for Kubernetes version 1.32 is deprecated and will be removed in kOps 1.38.

  • Support for AWS Classic Load Balancer (CLB) for the API is removed in kOps 1.37.

  • Support for containerd versions older than 2.2 (all end of life upstream) is deprecated and will be removed in kOps 1.38.

  • Support for gossip DNS, deprecated since kOps 1.29, is removed in kOps 1.37.

  • Support for Kopeio networking is removed in kOps 1.37.

  • The Spotinst integration is deprecated. kOps 1.38 will block the creation of new Spotinst clusters. kOps will temporarily support deletion of existing clusters for safe migration. kOps 1.39 will remove the integration. (#18717)

  • Cloud storage ACL support is deprecated. kOps will remove it in stages through 1.39. This deprecation includes KOPS_STATE_S3_ACL, GCS object ACL writes, the GoogleCloudBucketAcl feature flag, and the discovery-store public-read fallback. Use bucket policies or IAM instead. For AWS, use S3 Object Ownership. For GCS, use uniform bucket-level access. (#18716)