Release notes for kOps 1.21 series ¶
⚠ kOps 1.21 has not been released yet! ⚠
This is a document to gather the release notes prior to the release.
Significant changes ¶
Dedicated API Server nodes. ¶
kOps now supports extending the control plane with dedicated apiserver nodes. These nodes run in dedicated instance groups that can be scaled horizontally.
In 1.21, this feature is behind a feature flag as node role name, labels, taints, and domains can change based on feedback from the community.
Other significant changes ¶
- Protokube now runs as a systemd process rather than a docker container.
Breaking changes ¶
Required Actions ¶
Deprecations ¶
- Support for Kubernetes versions 1.15 and 1.16 are deprecated and will be removed in kOps 1.22.
- Support for launch configurations has been removed in favour of launch templates.
Full change list since 1.21.0 release ¶
1.21.0-alpha.2 to 1.21.0-alpha.1 ¶
- Release notes for 1.21.0-alpha.2 @hakman #10768
- Add troubleshooting of corrupted api server leases @olemarkus #10764
- Boot nodes without state store access @justinsb #10469
- Update GCE zones @bharath-123 #10771
- Kubetest2 - Use a shell lexer for passing extra args to
create cluster
@rifelpet #10772 - Use the kubeApiServerConfig clientCAFile field @slu2011 #10707
- Kubetest2 - Fix splitting of --create-args @rifelpet #10775
- Logging: don't suggest we are pre-creating DNS records unless we are @justinsb #10782
- Add missing versions to channels @olemarkus #10781
- fix: asset task copy docker image @johanneswuerbach #10767
- Add support for creating world-readable managedFiles @olemarkus #10778
- Update kubectl documentation with new flags @rpadovani,@hakman #10779
- Add overrides testing in lifecycle integration tests @rifelpet #10752
- Add AWS LoadBalancerController @olemarkus #10489
- Update Calico to v3.17.2 @hakman #10787
- Enable CSIMigrationAWS if CSI EBS driver is installed @olemarkus #10791
- Fill Role names in kops-controller-config instead of instance profile names when it is specified @h3poteto #10728
- Storage: Allow disabling of kOps's management of StorageClasses @seh #10733
- kubetest2 - Dump all pod logs in addition to host logs @rifelpet #10799
- Update Docker to v19.03.15 @hakman #10802
- Fix LaunchSpec TF output @hakman #10806
- Make protokube CP label setting consistent with kops-controller @olemarkus #10780
- Add deprecation notice for launch templates. @bharath-123 #10809
- add azure support for internal loadbalancer to k8s api @collin-woodruff-t1cg #10744
- Allow managed images for Azure instance groups @NickSchleicher #10797
- kubenet containerd: match upstream @justinsb #10759
- kubetest2: Add --host argument @justinsb #10814
- iptables: Use the lock when checking for existing rules @justinsb #10812
- Spotinst: Replace corev1.Taint to fix HCL2 serialization @liranp #10819
- Spotinst: Bump the Ocean Controller to 1.0.72 @liranp #10820
- Allow to control which subnets and IPs get used for the API loadbalancer @codablock #10741
- kubetest2: Call Test, not Execute @justinsb #10824
- Fix kdi 'must specify' error @olemarkus #10825
- Update aws-sdk-go @rifelpet #10830
- Use correct tag when creating node labels from azure cloud tags @NickSchleicher #10619
- Precreate the kops-controller DNS name @rifelpet #10833
- containerd installation: always configure, even if we don't install @justinsb #10813
- Release binaries for protokube and channels @hakman #10840
- Release 1.21.0-alpha.1 @hakman #10841
1.21.0-alpha.1 to 1.21.0-alpha.2 ¶
- Release notes for 1.21.0-alpha.1 @hakman #10844
- Update mock to v1.21.0-alpha.1 @hakman #10845
- Kubetest2 - terraform support @rifelpet #10697
- Actually enable systemd cgroup for containerd @codablock #10846
- Update Go to v1.15.8 @hakman #10853
- Add liveness probe for calico-kube-controllers @hakman #10856
- Fix OpenStack delete functions @ottosulin #10849
- Add support for CAS 1.20 + support for disabling CAS for a given IG @olemarkus #10857
- Bump aws node termination handler to 1.12.0 @bharath-123 #10863
- Kubetest2 - add ginkgo node debug logs @rifelpet #10866
- K8s Version Updates February 2021 @MoShitrit #10865
- Add note about remote identities @olemarkus #10868
- Bump metrics-server to 0.4.2 @olemarkus #10858
- kubetest2 - support terraform with
kops create cluster
@rifelpet #10867 - Add validation for instanceType and ami architecture @bharath-123 #10747
- Upgrade k8s 1.20 to latest patch version @MoShitrit #10875
- Update AWS CNI to latest patch version @MoShitrit #10876
- Fixes for 1.21 e2e tests @olemarkus #10879
- Release notes for 1.19.1 @justinsb #10883
- Improve machine type and image validation @hakman #10884
- fix loadBalancerID null pointer @collin-woodruff-t1cg #10886
- Update Openstack Cloud Go module to v1.20.1 @bmelbourne #10896
- Enforce 1.14 deprecation @olemarkus #10897
- add usage of subnet and routetable shared resources in azure @ngalantowicz #10900
- Update Calico to v3.18.0 @hakman #10904
- Adding Elastic IP Allocations to NLB API @timothyclarke #10872
- Release notes for 1.20.0-beta.1 @hakman #10909
- Update Google Cloud Go module to v0.77.0 @bmelbourne #10894
- Add Tagging to Instance Profiles and OIDC Providers @rifelpet #10832
- AWS LB controller is as of 1.20, not 1.19 @olemarkus #10919
- Spotinst: Prevent instance groups with the same suffix from being deleted @liranp #10918
- add support for azure public loadbalancer @collin-woodruff-t1cg #10915
- Fix nil pointer deference for image ID with spotinst @hakman #10924
- Update SSH documentation for ubuntu @jpugliesi #10931
- Fix no-schedule issue @christian-schlichtherle #10928
- Update Controller Runtime Go module to v0.8.2 @bmelbourne #10914
- Sort external policies when checking for changes @hakman #10940
- Instruct GH to collapse BUILD.bazel diffs by default @rifelpet #10912
- Further improve cloudLabel validation @olemarkus #10910
- Add a standardised set of labels on all resources @olemarkus #10796
- Bump external-dns to 0.7.6 @olemarkus #10946
- Update etcd-manager to 3.0.20210228 @justinsb #10949
- gce doesn't suffix the IG names with ClusterName @olemarkus #10944
- Add AWS Transit Gateway support @rifelpet #10948
- Fix node label conversion in Azure @kenji-cloudnatix #10935
- Spotinst: Bump the Ocean Controller to 1.0.73 @liranp #10960
- Spotinst: Don't skip LB attachments when SpotinstHybrid is enabled @liranp #10961
- Add explicit RBAC permissions for finalizers subresources @olemarkus #10966
- Fix typos in docs/getting_started @roim #10921
- Add support for CPU Credits on AWS t2 and t3 instance families @rifelpet #10934
- Add support for enable-cadvisor-json-endpoints with Kubelet @adrianmoisey #10957
- Exclude CP nodes from load balancers @olemarkus #10945
- Update k8s.io Go modules to v0.20.4 @bmelbourne #10965
- Update Go to v1.16 @bmelbourne #10892
- Add a note about informal office hours @olemarkus #10650
- Removing duplicate local and output values in terraform(#10786) @mmerrill3 #10978
- Add CloudLabels as --extra-tags to aws-ebs-csi driver @codablock #10976
- Use internal api url for jwks @olemarkus #10888
- Disable Calico Prometheus metrics by default @hakman #10982
- Add etcd-manager discoveryPollInterval option @ottosulin #10975
- Remove manually added labels from addons @hakman #10987
- Fix kops-controller rbac due to leader election change @olemarkus #10988
- Various cleanups around apply_cluster and awsmodel @olemarkus #10579
- Fix very minor formatting typos in docs/manifests_and_customizing_via_api @vitaliyf #10990
- Run protokube as a systemd service @bharath-123,@hakman #10574
- kubetest2 - don't overwrite create args that use equals signs @rifelpet #10994
- Remove support for launch configurations @bharath-123 #10937
- Use exponential backoff for DNS updates @hakman #10996
- Storage: Amend default choice for StorageClass management to honor a specified OpenStack-related value @seh #11002
- Add to 1.21 release notes @bharath-123 #11004
- Kubetest2 - Add support for publishing the kops version marker @rifelpet #11006
- Kubetest2 - Fix kops' --kubernetes-version with k8s version markers @rifelpet #11007
- Don't build kops during periodic upgrade tests @rifelpet #11005
- Remove extraneous field from integration test @rifelpet #11010
- Remove trailing newline from kubernetes version marker @rifelpet #11011
- aws: Graceful handling of EC2 detach errors @hwoarang #10740
- Kubetest2 - use same kops binary for all commands in upgrade scenario @rifelpet #11017
- Update Calico to v3.18.1 @hakman #11018
- Increase route53 retry count from 3 to 5 @rifelpet #11020
- Spotinst: Add support for block device mappings in Ocean Launch Spec @liranp #11009
- Allow cilium 1.10 @olemarkus #11026
- Fix rendering of multiple Docker insecure registries @hakman #11027
- azure: fix null pointer when updating in place cluster @collin-woodruff-t1cg #11015
- Release notes for 1.20.0-beta.2 @hakman #11034
- Update k8s dependencies to v1.21.0-beta.1 @hakman #11013
- Trim space on kops version markers @rifelpet #11037
- Honor OS update policy at InstanceGroup level too @seh #10913
- Update Go to v1.16.2 @hakman #11039
- Create an environment file for kops-configuration systemd process @bharath-123 #11042
- Improve instance type validation error message @bharath-123 #11043
- Revert upgrade script to build kops @rifelpet #11044
- cluster validation - allow flapping of validation errors @rifelpet #11049
- Update Terraform to v0.14.8 @bmelbourne #11051
- Cleanup some nodeup & protokube logging @rifelpet #11052
- Update Go modules to latest versions @bmelbourne #11047
- Add channels entries for image architecture @hakman #11046
- fix CNI bin path in troubleshoot.md @adrianmester #11061
- Kubetest2 - Add GCE default SSH key values from prow jobs @rifelpet #11065
- correct a word for readme @yojay11717 #11066
- Update Bazel to v3.5.0 @hakman #11041
- Install bazelisk before pushing images @hakman #11067
- Kubetest2 - Add boskos for GCE support @rifelpet #11070
- Download kubectl to /opt/kops/bin on Flatcar OS @rifelpet #11054
- Kubetest2 - initialize boskos heartbeat channel @rifelpet #11073
- Instance roles for service accounts (IRSA) contd @rifelpet,@olemarkus #10756
- Kubetest2 - add more validation time for --target terraform @rifelpet #11077
- Fix GCE channels version constraints @rifelpet #11076
- Update k8s versions with March 2021 releases @MoShitrit #11075
- Upgrade AWS CNI to version 1.7.10 @MoShitrit #11078
- Improve error messages around PublicJWKS @justinsb #11085
- Don't add control-plane DNS permissions with UseServiceAccountIAM @justinsb #11086
- Ensure a publicdatastore exists for jwks and that it can only be s3 @olemarkus #11081
- Apiserver nodes @olemarkus #10722
- fix(docs): cpuCFSQuotaPeriod needs a feature gate @danmx #11071
- Update Ubuntu 20.04 to latest AMI @bmelbourne #11083
- Re-add integration tests for jwks @justinsb #11087
- Replace go-bindata with go:embed @rifelpet #11089
- Dns controller fixes @olemarkus #11069
- Remove unused RoleLabelName16 @justinsb #11097
- Add additional IOPS validation for AWS EBS gp3 volumes @lichuan0620 #10843
- Update google SDK libraries @justinsb #11096
- Add values page @justinsb #11094
- Deeper validation in dns controller tests @justinsb #11095
- Ensure protokube can connect to kube-apiserver before starting the sync loop @olemarkus #11093
- Remove dbus dependency @bharath-123 #11082
- Have nodeup retry kops-controller bootstrapping sooner if DNS isn't setup @rifelpet #11101
- Update AWS zones used by e2e tests @rifelpet #11103
- Add docs about dedicated apiserver ndoes @olemarkus #11090
- Put awslbcontroller on the control-plane @olemarkus #11091
- Release 1.21.0 alpha.2 @hakman #11109
1.21.0-alpha.2 to 1.21.0-alpha.3 ¶
- Release notes for 1.21.0-alpha.2 @hakman #11111
- Update release process docs @hakman #11112
- Use "tag on create" for EIPs, NLBs, and TargetGroups @rifelpet #11107
- Load env vars from file for kops-configuration service @hakman #11114
- Update containerd to v1.3.10/v1.4.4 @bmelbourne #11084
- [DigitalOcean] Fix DO Tag issue @srikiz #11102
- Kubetest2 - Setup SSH keys for GCE @rifelpet #11123
- Validate that kube-apiserver has the necessary authz modes set @olemarkus #11127
- Remove instance-selector label @bharath-123 #11048
- Kubetest2 - fix temp directory created for GCE SSH keys @rifelpet #11133
- replace hard coded aws region checks with aws sdk calls @guydog28 #11119
- kubetest2 - Specify GCE network name @rifelpet #11139
- Update protokube systemd unit docs link @rifelpet #11138
- Add scaleDownDelayAfterAdd to clusterAutoscaler spec @jurriaanpro #11140
- Update cluster_spec.md @carnivorelogic #11142
- minor protokube code clean up @bharath-123 #11143
- Pass ctx to drain helper @olemarkus #11146
- Change registrable domains to placeholders @lukehinds #11147
- Add tags to instance profile and OIDC provider terraform resources @rifelpet #11149
- Clarify release notes around exporting kubeconfig @justinsb #11154
- Expand flag help on --user flags @justinsb #11153
- Update Getting Started AWS guide @allir #11150
- fix the mistake link in addons.md @maoyangLiu #11151
- cloudbuild: capture some hashes @justinsb #11159
- Only update kops-controller pods on deletion @olemarkus #10871
- Side load images also on apiserver @olemarkus #11156
- Add an option to skip NTP installation @kenji-cloudnatix #11160
- kubetest2 - Pass GOPATH when building kops @rifelpet #11167
- Filter kOps NatGateways from route table @zetaab #11169
- Bump k8s deps to 1.21-rc.0 @olemarkus #11168
- Allow setting dedicated apiserver node count from create cluster cmd @olemarkus #11152
- Update Go to v1.16.3 @bmelbourne #11174
- Add integration test for aws lb controller @olemarkus #11175
- Enable use of irsa for aws load balancer controller @olemarkus #11088
- Increase timeout and update images for postsubmit job @rifelpet #11177
- Update Go modules to latest versions @bmelbourne #11176
- Kubetest2 - Add flag to expose cluster validation wait time @rifelpet #11178
- Spotinst: Use BDM to configure the root volume size at VNG level @liranp #11179
- Spotinst: Configure headroom resources only at the VNG level @liranp #11181
- Update k8s dependencies to v1.21.0 @hakman #11188
- Release notes for 1.19.2 @justinsb #11193
- Update node local dns cache @zetaab #11057
- Update cilium.md @recollir #11189
- Release notes for 1.20.0 @justinsb #11196
- Docs: Remove 'prerelease' warning from 1.20 @justinsb #11198
- Kubetest2 - Create project-specific state store buckets in GCP @rifelpet #11200
- Update release compatibility matrix @johngmyers #11201
- Update integration tests to k8s v1.21.0 @bmelbourne #11206
- Kubetest2 - Set KOPS_BASE_URL to --build's stage location @rifelpet #11210
- Update Docker to v20.10.5 @bmelbourne #11195
- Rename the service account key @johngmyers #11207
- Update go deps @zetaab #11208
- Kubetest2 - detect errors creating GCS bucket @rifelpet #11212
- Kubetest2 - Ensure the bucket path is the final gsutil arg @rifelpet #11215
- Update IG tutorial for per-AZ node groups @rifelpet #11218
- Use "string" for architecture type in ChannelRecommendedImage @hakman #11220
- Always secure api -> kubelet communication @olemarkus #11185
- Fix etcd volume validation logic @hakman #11225
- Replace k8s.io/utils/mount with k8s.io/mount-utils @hakman #11229
- Release 1.21.0-alpha.3 @hakman #11231