Release notes for kOps 1.21 series

⚠ kOps 1.21 has not been released yet! ⚠

This is a document to gather the release notes prior to the release.

Significant changes

Dedicated API Server nodes.

kOps now supports extending the control plane with dedicated apiserver nodes. These nodes run in dedicated instance groups that can be scaled horizontally.

In 1.21, this feature is behind a feature flag as node role name, labels, taints, and domains can change based on feedback from the community.

Other significant changes

• Protokube now runs as a systemd process rather than a docker container.

Breaking changes

Required Actions

Deprecations

• Support for Kubernetes versions 1.15 and 1.16 are deprecated and will be removed in kOps 1.22.
• Support for launch configurations has been removed in favour of launch templates.

Full change list since 1.21.0 release

1.21.0-alpha.2 to 1.21.0-alpha.1

• Release notes for 1.21.0-alpha.2 @hakman #10768
• Add troubleshooting of corrupted api server leases @olemarkus #10764
• Boot nodes without state store access @justinsb #10469
• Update GCE zones @bharath-123 #10771
• Kubetest2 - Use a shell lexer for passing extra args to create cluster @rifelpet #10772
• Use the kubeApiServerConfig clientCAFile field @slu2011 #10707
• Kubetest2 - Fix splitting of --create-args @rifelpet #10775
• Logging: don't suggest we are pre-creating DNS records unless we are @justinsb #10782
• Add missing versions to channels @olemarkus #10781
• fix: asset task copy docker image @johanneswuerbach #10767
• Add support for creating world-readable managedFiles @olemarkus #10778
• Update kubectl documentation with new flags @rpadovani,@hakman #10779
• Add overrides testing in lifecycle integration tests @rifelpet #10752
• Add AWS LoadBalancerController @olemarkus #10489
• Update Calico to v3.17.2 @hakman #10787
• Enable CSIMigrationAWS if CSI EBS driver is installed @olemarkus #10791
• Fill Role names in kops-controller-config instead of instance profile names when it is specified @h3poteto #10728
• Storage: Allow disabling of kOps's management of StorageClasses @seh #10733
• kubetest2 - Dump all pod logs in addition to host logs @rifelpet #10799
• Update Docker to v19.03.15 @hakman #10802
• Fix LaunchSpec TF output @hakman #10806
• Make protokube CP label setting consistent with kops-controller @olemarkus #10780
• Add deprecation notice for launch templates. @bharath-123 #10809
• add azure support for internal loadbalancer to k8s api @collin-woodruff-t1cg #10744
• Allow managed images for Azure instance groups @NickSchleicher #10797
• kubenet containerd: match upstream @justinsb #10759
• kubetest2: Add --host argument @justinsb #10814
• iptables: Use the lock when checking for existing rules @justinsb #10812
• Spotinst: Replace corev1.Taint to fix HCL2 serialization @liranp #10819
• Spotinst: Bump the Ocean Controller to 1.0.72 @liranp #10820
• Allow to control which subnets and IPs get used for the API loadbalancer @codablock #10741
• kubetest2: Call Test, not Execute @justinsb #10824
• Fix kdi 'must specify' error @olemarkus #10825
• Update aws-sdk-go @rifelpet #10830
• Use correct tag when creating node labels from azure cloud tags @NickSchleicher #10619
• Precreate the kops-controller DNS name @rifelpet #10833
• containerd installation: always configure, even if we don't install @justinsb #10813
• Release binaries for protokube and channels @hakman #10840
• Release 1.21.0-alpha.1 @hakman #10841

1.21.0-alpha.1 to 1.21.0-alpha.2

• Release notes for 1.21.0-alpha.1 @hakman #10844
• Update mock to v1.21.0-alpha.1 @hakman #10845
• Kubetest2 - terraform support @rifelpet #10697
• Actually enable systemd cgroup for containerd @codablock #10846
• Update Go to v1.15.8 @hakman #10853
• Add liveness probe for calico-kube-controllers @hakman #10856
• Fix OpenStack delete functions @ottosulin #10849
• Add support for CAS 1.20 + support for disabling CAS for a given IG @olemarkus #10857
• Bump aws node termination handler to 1.12.0 @bharath-123 #10863
• Kubetest2 - add ginkgo node debug logs @rifelpet #10866
• K8s Version Updates February 2021 @MoShitrit #10865
• Add note about remote identities @olemarkus #10868
• Bump metrics-server to 0.4.2 @olemarkus #10858
• kubetest2 - support terraform with kops create cluster @rifelpet #10867
• Add validation for instanceType and ami architecture @bharath-123 #10747
• Upgrade k8s 1.20 to latest patch version @MoShitrit #10875
• Update AWS CNI to latest patch version @MoShitrit #10876
• Fixes for 1.21 e2e tests @olemarkus #10879
• Release notes for 1.19.1 @justinsb #10883
• Improve machine type and image validation @hakman #10884
• fix loadBalancerID null pointer @collin-woodruff-t1cg #10886
• Update Openstack Cloud Go module to v1.20.1 @bmelbourne #10896
• Enforce 1.14 deprecation @olemarkus #10897
• add usage of subnet and routetable shared resources in azure @ngalantowicz #10900
• Update Calico to v3.18.0 @hakman #10904
• Adding Elastic IP Allocations to NLB API @timothyclarke #10872
• Release notes for 1.20.0-beta.1 @hakman #10909
• Update Google Cloud Go module to v0.77.0 @bmelbourne #10894
• Add Tagging to Instance Profiles and OIDC Providers @rifelpet #10832
• AWS LB controller is as of 1.20, not 1.19 @olemarkus #10919
• Spotinst: Prevent instance groups with the same suffix from being deleted @liranp #10918
• add support for azure public loadbalancer @collin-woodruff-t1cg #10915
• Fix nil pointer deference for image ID with spotinst @hakman #10924
• Update SSH documentation for ubuntu @jpugliesi #10931
• Fix no-schedule issue @christian-schlichtherle #10928
• Update Controller Runtime Go module to v0.8.2 @bmelbourne #10914
• Sort external policies when checking for changes @hakman #10940
• Instruct GH to collapse BUILD.bazel diffs by default @rifelpet #10912
• Further improve cloudLabel validation @olemarkus #10910
• Add a standardised set of labels on all resources @olemarkus #10796
• Bump external-dns to 0.7.6 @olemarkus #10946
• Update etcd-manager to 3.0.20210228 @justinsb #10949
• gce doesn't suffix the IG names with ClusterName @olemarkus #10944
• Add AWS Transit Gateway support @rifelpet #10948
• Fix node label conversion in Azure @kenji-cloudnatix #10935
• Spotinst: Bump the Ocean Controller to 1.0.73 @liranp #10960
• Spotinst: Don't skip LB attachments when SpotinstHybrid is enabled @liranp #10961
• Add explicit RBAC permissions for finalizers subresources @olemarkus #10966
• Fix typos in docs/getting_started @roim #10921
• Add support for CPU Credits on AWS t2 and t3 instance families @rifelpet #10934
• Add support for enable-cadvisor-json-endpoints with Kubelet @adrianmoisey #10957
• Exclude CP nodes from load balancers @olemarkus #10945
• Update k8s.io Go modules to v0.20.4 @bmelbourne #10965
• Update Go to v1.16 @bmelbourne #10892
• Add a note about informal office hours @olemarkus #10650
• Removing duplicate local and output values in terraform(#10786) @mmerrill3 #10978
• Add CloudLabels as --extra-tags to aws-ebs-csi driver @codablock #10976
• Use internal api url for jwks @olemarkus #10888
• Disable Calico Prometheus metrics by default @hakman #10982
• Add etcd-manager discoveryPollInterval option @ottosulin #10975
• Remove manually added labels from addons @hakman #10987
• Fix kops-controller rbac due to leader election change @olemarkus #10988
• Various cleanups around apply_cluster and awsmodel @olemarkus #10579
• Fix very minor formatting typos in docs/manifests_and_customizing_via_api @vitaliyf #10990
• Run protokube as a systemd service @bharath-123,@hakman #10574
• kubetest2 - don't overwrite create args that use equals signs @rifelpet #10994
• Remove support for launch configurations @bharath-123 #10937
• Use exponential backoff for DNS updates @hakman #10996
• Storage: Amend default choice for StorageClass management to honor a specified OpenStack-related value @seh #11002
• Add to 1.21 release notes @bharath-123 #11004
• Kubetest2 - Add support for publishing the kops version marker @rifelpet #11006
• Kubetest2 - Fix kops' --kubernetes-version with k8s version markers @rifelpet #11007
• Don't build kops during periodic upgrade tests @rifelpet #11005
• Remove extraneous field from integration test @rifelpet #11010
• Remove trailing newline from kubernetes version marker @rifelpet #11011
• aws: Graceful handling of EC2 detach errors @hwoarang #10740
• Kubetest2 - use same kops binary for all commands in upgrade scenario @rifelpet #11017
• Update Calico to v3.18.1 @hakman #11018
• Increase route53 retry count from 3 to 5 @rifelpet #11020
• Spotinst: Add support for block device mappings in Ocean Launch Spec @liranp #11009
• Allow cilium 1.10 @olemarkus #11026
• Fix rendering of multiple Docker insecure registries @hakman #11027
• azure: fix null pointer when updating in place cluster @collin-woodruff-t1cg #11015
• Release notes for 1.20.0-beta.2 @hakman #11034
• Update k8s dependencies to v1.21.0-beta.1 @hakman #11013
• Trim space on kops version markers @rifelpet #11037
• Honor OS update policy at InstanceGroup level too @seh #10913
• Update Go to v1.16.2 @hakman #11039
• Create an environment file for kops-configuration systemd process @bharath-123 #11042
• Improve instance type validation error message @bharath-123 #11043
• Revert upgrade script to build kops @rifelpet #11044
• cluster validation - allow flapping of validation errors @rifelpet #11049
• Update Terraform to v0.14.8 @bmelbourne #11051
• Cleanup some nodeup & protokube logging @rifelpet #11052
• Update Go modules to latest versions @bmelbourne #11047
• Add channels entries for image architecture @hakman #11046
• fix CNI bin path in troubleshoot.md @adrianmester #11061
• Kubetest2 - Add GCE default SSH key values from prow jobs @rifelpet #11065
• correct a word for readme @yojay11717 #11066
• Update Bazel to v3.5.0 @hakman #11041
• Install bazelisk before pushing images @hakman #11067
• Kubetest2 - Add boskos for GCE support @rifelpet #11070
• Download kubectl to /opt/kops/bin on Flatcar OS @rifelpet #11054
• Kubetest2 - initialize boskos heartbeat channel @rifelpet #11073
• Instance roles for service accounts (IRSA) contd @rifelpet,@olemarkus #10756
• Kubetest2 - add more validation time for --target terraform @rifelpet #11077
• Fix GCE channels version constraints @rifelpet #11076
• Update k8s versions with March 2021 releases @MoShitrit #11075
• Upgrade AWS CNI to version 1.7.10 @MoShitrit #11078
• Improve error messages around PublicJWKS @justinsb #11085
• Don't add control-plane DNS permissions with UseServiceAccountIAM @justinsb #11086
• Ensure a publicdatastore exists for jwks and that it can only be s3 @olemarkus #11081
• Apiserver nodes @olemarkus #10722
• fix(docs): cpuCFSQuotaPeriod needs a feature gate @danmx #11071
• Update Ubuntu 20.04 to latest AMI @bmelbourne #11083
• Re-add integration tests for jwks @justinsb #11087
• Replace go-bindata with go:embed @rifelpet #11089
• Dns controller fixes @olemarkus #11069
• Remove unused RoleLabelName16 @justinsb #11097
• Add additional IOPS validation for AWS EBS gp3 volumes @lichuan0620 #10843
• Update google SDK libraries @justinsb #11096
• Add values page @justinsb #11094
• Deeper validation in dns controller tests @justinsb #11095
• Ensure protokube can connect to kube-apiserver before starting the sync loop @olemarkus #11093
• Remove dbus dependency @bharath-123 #11082
• Have nodeup retry kops-controller bootstrapping sooner if DNS isn't setup @rifelpet #11101
• Update AWS zones used by e2e tests @rifelpet #11103
• Add docs about dedicated apiserver ndoes @olemarkus #11090
• Put awslbcontroller on the control-plane @olemarkus #11091
• Release 1.21.0 alpha.2 @hakman #11109

1.21.0-alpha.2 to 1.21.0-alpha.3

• Release notes for 1.21.0-alpha.2 @hakman #11111
• Update release process docs @hakman #11112
• Use "tag on create" for EIPs, NLBs, and TargetGroups @rifelpet #11107
• Load env vars from file for kops-configuration service @hakman #11114
• Update containerd to v1.3.10/v1.4.4 @bmelbourne #11084
• [DigitalOcean] Fix DO Tag issue @srikiz #11102
• Kubetest2 - Setup SSH keys for GCE @rifelpet #11123
• Validate that kube-apiserver has the necessary authz modes set @olemarkus #11127
• Remove instance-selector label @bharath-123 #11048
• Kubetest2 - fix temp directory created for GCE SSH keys @rifelpet #11133
• replace hard coded aws region checks with aws sdk calls @guydog28 #11119
• kubetest2 - Specify GCE network name @rifelpet #11139
• Update protokube systemd unit docs link @rifelpet #11138
• Add scaleDownDelayAfterAdd to clusterAutoscaler spec @jurriaanpro #11140
• Update cluster_spec.md @carnivorelogic #11142
• minor protokube code clean up @bharath-123 #11143
• Pass ctx to drain helper @olemarkus #11146
• Change registrable domains to placeholders @lukehinds #11147
• Add tags to instance profile and OIDC provider terraform resources @rifelpet #11149
• Clarify release notes around exporting kubeconfig @justinsb #11154
• Expand flag help on --user flags @justinsb #11153
• Update Getting Started AWS guide @allir #11150
• fix the mistake link in addons.md @maoyangLiu #11151
• cloudbuild: capture some hashes @justinsb #11159
• Only update kops-controller pods on deletion @olemarkus #10871
• Side load images also on apiserver @olemarkus #11156
• Add an option to skip NTP installation @kenji-cloudnatix #11160
• kubetest2 - Pass GOPATH when building kops @rifelpet #11167
• Filter kOps NatGateways from route table @zetaab #11169
• Bump k8s deps to 1.21-rc.0 @olemarkus #11168
• Allow setting dedicated apiserver node count from create cluster cmd @olemarkus #11152
• Update Go to v1.16.3 @bmelbourne #11174
• Add integration test for aws lb controller @olemarkus #11175
• Enable use of irsa for aws load balancer controller @olemarkus #11088
• Increase timeout and update images for postsubmit job @rifelpet #11177
• Update Go modules to latest versions @bmelbourne #11176
• Kubetest2 - Add flag to expose cluster validation wait time @rifelpet #11178
• Spotinst: Use BDM to configure the root volume size at VNG level @liranp #11179
• Spotinst: Configure headroom resources only at the VNG level @liranp #11181
• Update k8s dependencies to v1.21.0 @hakman #11188
• Release notes for 1.19.2 @justinsb #11193
• Update node local dns cache @zetaab #11057
• Update cilium.md @recollir #11189
• Release notes for 1.20.0 @justinsb #11196
• Docs: Remove 'prerelease' warning from 1.20 @justinsb #11198
• Kubetest2 - Create project-specific state store buckets in GCP @rifelpet #11200
• Update release compatibility matrix @johngmyers #11201
• Update integration tests to k8s v1.21.0 @bmelbourne #11206
• Kubetest2 - Set KOPS_BASE_URL to --build's stage location @rifelpet #11210
• Update Docker to v20.10.5 @bmelbourne #11195
• Rename the service account key @johngmyers #11207
• Update go deps @zetaab #11208
• Kubetest2 - detect errors creating GCS bucket @rifelpet #11212
• Kubetest2 - Ensure the bucket path is the final gsutil arg @rifelpet #11215
• Update IG tutorial for per-AZ node groups @rifelpet #11218
• Use "string" for architecture type in ChannelRecommendedImage @hakman #11220
• Always secure api -> kubelet communication @olemarkus #11185
• Fix etcd volume validation logic @hakman #11225
• Replace k8s.io/utils/mount with k8s.io/mount-utils @hakman #11229
• Release 1.21.0-alpha.3 @hakman #11231