Fixing Missing TKRs in Existing TKGS Deployments
I regularly check the Tanzu Kubernetes Releases (TKR) release notes page for new updates. Yesterday, a new TKR was released with support for Kubernetes 1.28.8, and while attempting to test this new version in my TKGS environment, I realized that the TKR was not present in my environment and I started wondering why, as normally, when new TKRs are released, they immediately become available for deployment, since the vCenter is subscribed to the VMware public content library where all the TKRs are hosted. This time, that was not the case, so I started investigating.
The first place to look at is the output of the kubectl get tkr -A command. In this case, there was nothing for Kubernetes 1.28.x.
$ kubectl get tkr -A
NAME VERSION READY COMPATIBLE CREATED
v1.16.12---vmware.1-tkg.1.da7afe7 v1.16.12+vmware.1-tkg.1.da7afe7 False False 41d
v1.16.14---vmware.1-tkg.1.ada4837 v1.16.14+vmware.1-tkg.1.ada4837 False False 41d
v1.16.8---vmware.1-tkg.3.60d2ffd v1.16.8+vmware.1-tkg.3.60d2ffd False False 41d
v1.17.11---vmware.1-tkg.1.15f1e18 v1.17.11+vmware.1-tkg.1.15f1e18 False False 41d
v1.17.11---vmware.1-tkg.2.ad3d374 v1.17.11+vmware.1-tkg.2.ad3d374 False False 41d
v1.17.13---vmware.1-tkg.2.2c133ed v1.17.13+vmware.1-tkg.2.2c133ed False False 41d
v1.17.17---vmware.1-tkg.1.d44d45a v1.17.17+vmware.1-tkg.1.d44d45a False False 41d
v1.17.7---vmware.1-tkg.1.154236c v1.17.7+vmware.1-tkg.1.154236c False False 41d
v1.17.8---vmware.1-tkg.1.5417466 v1.17.8+vmware.1-tkg.1.5417466 False False 41d
v1.18.10---vmware.1-tkg.1.3a6cd48 v1.18.10+vmware.1-tkg.1.3a6cd48 False False 41d
v1.18.15---vmware.1-tkg.1.600e412 v1.18.15+vmware.1-tkg.1.600e412 False False 41d
v1.18.15---vmware.1-tkg.2.ebf6117 v1.18.15+vmware.1-tkg.2.ebf6117 False False 41d
v1.18.19---vmware.1-tkg.1.17af790 v1.18.19+vmware.1-tkg.1.17af790 False False 41d
v1.18.5---vmware.1-tkg.1.c40d30d v1.18.5+vmware.1-tkg.1.c40d30d False False 41d
v1.19.11---vmware.1-tkg.1.9d9b236 v1.19.11+vmware.1-tkg.1.9d9b236 False False 41d
v1.19.14---vmware.1-tkg.1.8753786 v1.19.14+vmware.1-tkg.1.8753786 False False 41d
v1.19.16---vmware.1-tkg.1.df910e2 v1.19.16+vmware.1-tkg.1.df910e2 False False 41d
v1.19.7---vmware.1-tkg.1.fc82c41 v1.19.7+vmware.1-tkg.1.fc82c41 False False 41d
v1.19.7---vmware.1-tkg.2.f52f85a v1.19.7+vmware.1-tkg.2.f52f85a False False 41d
v1.20.12---vmware.1-tkg.1.b9a42f3 v1.20.12+vmware.1-tkg.1.b9a42f3 False False 41d
v1.20.2---vmware.1-tkg.1.1d4f79a v1.20.2+vmware.1-tkg.1.1d4f79a False False 41d
v1.20.2---vmware.1-tkg.2.3e10706 v1.20.2+vmware.1-tkg.2.3e10706 False False 41d
v1.20.7---vmware.1-tkg.1.7fb9067 v1.20.7+vmware.1-tkg.1.7fb9067 False False 41d
v1.20.8---vmware.1-tkg.2 v1.20.8+vmware.1-tkg.2 False False 41d
v1.20.9---vmware.1-tkg.1.a4cee5b v1.20.9+vmware.1-tkg.1.a4cee5b False False 41d
v1.21.2---vmware.1-tkg.1.ee25d55 v1.21.2+vmware.1-tkg.1.ee25d55 False False 41d
v1.21.6---vmware.1-tkg.1 v1.21.6+vmware.1-tkg.1 False False 41d
v1.21.6---vmware.1-tkg.1.b3d708a v1.21.6+vmware.1-tkg.1.b3d708a False False 41d
v1.22.9---vmware.1-tkg.1 v1.22.9+vmware.1-tkg.1 False False 41d
v1.22.9---vmware.1-tkg.1.cc71bc8 v1.22.9+vmware.1-tkg.1.cc71bc8 False False 41d
v1.23.15---vmware.1-tkg.4 v1.23.15+vmware.1-tkg.4 True True 41d
v1.23.8---vmware.2-tkg.2-zshippable v1.23.8+vmware.2-tkg.2-zshippable True True 41d
v1.23.8---vmware.3-tkg.1 v1.23.8+vmware.3-tkg.1 True True 41d
v1.23.8---vmware.3-tkg.1.ubuntu v1.23.8+vmware.3-tkg.1.ubuntu True True 41d
v1.24.11---vmware.1-fips.1-tkg.1 v1.24.11+vmware.1-fips.1-tkg.1 True True 41d
v1.24.11---vmware.1-fips.1-tkg.1.ubuntu v1.24.11+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.24.9---vmware.1-tkg.4 v1.24.9+vmware.1-tkg.4 True True 41d
v1.25.13---vmware.1-fips.1-tkg.1 v1.25.13+vmware.1-fips.1-tkg.1 True True 41d
v1.25.13---vmware.1-fips.1-tkg.1.ubuntu v1.25.13+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.25.7---vmware.3-fips.1-tkg.1 v1.25.7+vmware.3-fips.1-tkg.1 True True 41d
v1.26.10---vmware.1-fips.1-tkg.1 v1.26.10+vmware.1-fips.1-tkg.1 True True 41d
v1.26.10---vmware.1-fips.1-tkg.1.ubuntu v1.26.10+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.26.12---vmware.2-fips.1-tkg.2 v1.26.12+vmware.2-fips.1-tkg.2 True True 41d
v1.26.12---vmware.2-fips.1-tkg.2.ubuntu v1.26.12+vmware.2-fips.1-tkg.2.ubuntu True True 41d
v1.26.13---vmware.1-fips.1-tkg.3 v1.26.13+vmware.1-fips.1-tkg.3 True True 41d
v1.26.5---vmware.2-fips.1-tkg.1 v1.26.5+vmware.2-fips.1-tkg.1 True True 41d
v1.27.10---vmware.1-fips.1-tkg.1 v1.27.10+vmware.1-fips.1-tkg.1 True True 33d
v1.27.10---vmware.1-fips.1-tkg.1.ubuntu v1.27.10+vmware.1-fips.1-tkg.1.ubuntu True True 32d
v1.27.11---vmware.1-fips.1-tkg.2 v1.27.11+vmware.1-fips.1-tkg.2 True True 15d
v1.27.6---vmware.1-fips.1-tkg.1 v1.27.6+vmware.1-fips.1-tkg.1 True True 41d
v1.27.6---vmware.1-fips.1-tkg.1.ubuntu v1.27.6+vmware.1-fips.1-tkg.1.ubuntu True True 41d
Knowing that the TKR data is all taken from the content library, I triggered a sync for the Kubernetes Service Content Library from the vCenter UI.
I got the following error message:
A general system error occurred: HTTP request error: cannot authenticate SSL certificate for host wp-content.vmware.com.
I copied the content library URL and opened it in a web browser so I could look at the certificate information since it is not available from the vCenter UI.
I then realized the certificate had been updated.
As you can see, this certificate is now signed by Broadcom Inc. and was issued on 05/05/2024.
I then opened the content library settings.
And applied it by hitting OK without making any changes to the configuration.
The new certificate thumbprint was then retrieved, and all I had to do was trust it and proceed.
I then triggered the sync again, which was now successful.
Finally, I went back to the terminal and ran the kubectl get tkr -A command once again.
The new TKR was now listed.
$ kubectl get tkr -A
NAME VERSION READY COMPATIBLE CREATED
v1.16.12---vmware.1-tkg.1.da7afe7 v1.16.12+vmware.1-tkg.1.da7afe7 False False 41d
v1.16.14---vmware.1-tkg.1.ada4837 v1.16.14+vmware.1-tkg.1.ada4837 False False 41d
v1.16.8---vmware.1-tkg.3.60d2ffd v1.16.8+vmware.1-tkg.3.60d2ffd False False 41d
v1.17.11---vmware.1-tkg.1.15f1e18 v1.17.11+vmware.1-tkg.1.15f1e18 False False 41d
v1.17.11---vmware.1-tkg.2.ad3d374 v1.17.11+vmware.1-tkg.2.ad3d374 False False 41d
v1.17.13---vmware.1-tkg.2.2c133ed v1.17.13+vmware.1-tkg.2.2c133ed False False 41d
v1.17.17---vmware.1-tkg.1.d44d45a v1.17.17+vmware.1-tkg.1.d44d45a False False 41d
v1.17.7---vmware.1-tkg.1.154236c v1.17.7+vmware.1-tkg.1.154236c False False 41d
v1.17.8---vmware.1-tkg.1.5417466 v1.17.8+vmware.1-tkg.1.5417466 False False 41d
v1.18.10---vmware.1-tkg.1.3a6cd48 v1.18.10+vmware.1-tkg.1.3a6cd48 False False 41d
v1.18.15---vmware.1-tkg.1.600e412 v1.18.15+vmware.1-tkg.1.600e412 False False 41d
v1.18.15---vmware.1-tkg.2.ebf6117 v1.18.15+vmware.1-tkg.2.ebf6117 False False 41d
v1.18.19---vmware.1-tkg.1.17af790 v1.18.19+vmware.1-tkg.1.17af790 False False 41d
v1.18.5---vmware.1-tkg.1.c40d30d v1.18.5+vmware.1-tkg.1.c40d30d False False 41d
v1.19.11---vmware.1-tkg.1.9d9b236 v1.19.11+vmware.1-tkg.1.9d9b236 False False 41d
v1.19.14---vmware.1-tkg.1.8753786 v1.19.14+vmware.1-tkg.1.8753786 False False 41d
v1.19.16---vmware.1-tkg.1.df910e2 v1.19.16+vmware.1-tkg.1.df910e2 False False 41d
v1.19.7---vmware.1-tkg.1.fc82c41 v1.19.7+vmware.1-tkg.1.fc82c41 False False 41d
v1.19.7---vmware.1-tkg.2.f52f85a v1.19.7+vmware.1-tkg.2.f52f85a False False 41d
v1.20.12---vmware.1-tkg.1.b9a42f3 v1.20.12+vmware.1-tkg.1.b9a42f3 False False 41d
v1.20.2---vmware.1-tkg.1.1d4f79a v1.20.2+vmware.1-tkg.1.1d4f79a False False 41d
v1.20.2---vmware.1-tkg.2.3e10706 v1.20.2+vmware.1-tkg.2.3e10706 False False 41d
v1.20.7---vmware.1-tkg.1.7fb9067 v1.20.7+vmware.1-tkg.1.7fb9067 False False 41d
v1.20.8---vmware.1-tkg.2 v1.20.8+vmware.1-tkg.2 False False 41d
v1.20.9---vmware.1-tkg.1.a4cee5b v1.20.9+vmware.1-tkg.1.a4cee5b False False 41d
v1.21.2---vmware.1-tkg.1.ee25d55 v1.21.2+vmware.1-tkg.1.ee25d55 False False 41d
v1.21.6---vmware.1-tkg.1 v1.21.6+vmware.1-tkg.1 False False 41d
v1.21.6---vmware.1-tkg.1.b3d708a v1.21.6+vmware.1-tkg.1.b3d708a False False 41d
v1.22.9---vmware.1-tkg.1 v1.22.9+vmware.1-tkg.1 False False 41d
v1.22.9---vmware.1-tkg.1.cc71bc8 v1.22.9+vmware.1-tkg.1.cc71bc8 False False 41d
v1.23.15---vmware.1-tkg.4 v1.23.15+vmware.1-tkg.4 True True 41d
v1.23.8---vmware.2-tkg.2-zshippable v1.23.8+vmware.2-tkg.2-zshippable True True 41d
v1.23.8---vmware.3-tkg.1 v1.23.8+vmware.3-tkg.1 True True 41d
v1.23.8---vmware.3-tkg.1.ubuntu v1.23.8+vmware.3-tkg.1.ubuntu True True 41d
v1.24.11---vmware.1-fips.1-tkg.1 v1.24.11+vmware.1-fips.1-tkg.1 True True 41d
v1.24.11---vmware.1-fips.1-tkg.1.ubuntu v1.24.11+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.24.9---vmware.1-tkg.4 v1.24.9+vmware.1-tkg.4 True True 41d
v1.25.13---vmware.1-fips.1-tkg.1 v1.25.13+vmware.1-fips.1-tkg.1 True True 41d
v1.25.13---vmware.1-fips.1-tkg.1.ubuntu v1.25.13+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.25.7---vmware.3-fips.1-tkg.1 v1.25.7+vmware.3-fips.1-tkg.1 True True 41d
v1.26.10---vmware.1-fips.1-tkg.1 v1.26.10+vmware.1-fips.1-tkg.1 True True 41d
v1.26.10---vmware.1-fips.1-tkg.1.ubuntu v1.26.10+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.26.12---vmware.2-fips.1-tkg.2 v1.26.12+vmware.2-fips.1-tkg.2 True True 41d
v1.26.12---vmware.2-fips.1-tkg.2.ubuntu v1.26.12+vmware.2-fips.1-tkg.2.ubuntu True True 41d
v1.26.13---vmware.1-fips.1-tkg.3 v1.26.13+vmware.1-fips.1-tkg.3 True True 41d
v1.26.5---vmware.2-fips.1-tkg.1 v1.26.5+vmware.2-fips.1-tkg.1 True True 41d
v1.27.10---vmware.1-fips.1-tkg.1 v1.27.10+vmware.1-fips.1-tkg.1 True True 33d
v1.27.10---vmware.1-fips.1-tkg.1.ubuntu v1.27.10+vmware.1-fips.1-tkg.1.ubuntu True True 32d
v1.27.11---vmware.1-fips.1-tkg.2 v1.27.11+vmware.1-fips.1-tkg.2 True True 15d
v1.27.6---vmware.1-fips.1-tkg.1 v1.27.6+vmware.1-fips.1-tkg.1 True True 41d
v1.27.6---vmware.1-fips.1-tkg.1.ubuntu v1.27.6+vmware.1-fips.1-tkg.1.ubuntu True True 41d
v1.28.8---vmware.1-fips.1-tkg.2 v1.28.8+vmware.1-fips.1-tkg.2 True True 27s
This issue will occur in existing environments every time the content library endpoint certificate is updated. However, the fix is fairly quick and straightforward.