I recently had to demonstrate Tanzu Kubernetes Grid and its GPU integration capabilities. Developing a good use case and assembling the demo required some preliminary research.
During my research, I reached out to Jay Vyas, staff engineer at VMware, SIG Windows lead for Kubernetes, a Kubernetes legend, and an awesome guy in general. :) For those who don’t know Jay, he is also one of the authors of the fantastic book Core Kubernetes (look it up!).
Over the years of working with Tanzu Kubernetes Grid (TKG), one tool has stood out as a game-changer for resource customization: Carvel’s ytt. Whether tailoring cluster manifests, customizing TKG packages, or addressing unique deployment requirements, ytt has consistently been a fundamental part of the workflow. Its flexibility, power, and declarative approach make it an essential tool for anyone working deeply with Kubernetes in a TKG ecosystem.
But what exactly is ytt? Short for YAML Templating Tool, ytt is part of the Carvel suite of tools designed for Kubernetes resource management. It provides a powerful, programmable approach to templating YAML configurations by combining straightforward data values, overlays, and scripting capabilities. Unlike many traditional templating tools, ytt prioritizes structure and intent, making it easier to maintain, validate, and debug configurations—particularly in complex, large-scale Kubernetes environments.
I recently ran into an issue where TKGm had suddenly failed to connect to the vCenter server.
The issue turned out to be TLS-related, and I noticed that the vCenter server certificate had been replaced…
Due to the certificate issue, Cluster API components failed to communicate with vSphere, causing cluster reconciliation to fail, among other vSphere-related operations.
Since all TKG clusters in the environment were deployed with the VSPHERE_TLS_THUMBPRINT parameter specified, replacing the vCenter certificate breaks the connection to vSphere, as the TLS thumbprint changes as well.
Fluent-Bit is currently the preferred option for log shipping in TKG and is provided out of the box as a Tanzu package that can be easily deployed on each TKG/Kubernetes cluster.
A recent implementation required shipping all Kubernetes logs to Elasticsearch, complying with a specific naming convention for the Elasticsearch indices.
Applying such customizations requires you to utilize the Lua filter. Using the Lua filter, you can modify incoming records by invoking custom scripts to apply your logic when processing the records.
Continue reading
In a recent TKG implementation, it was required to configure Harbor with LDAPS rather than LDAP.
I deployed the Harbor package on the TKG shared services cluster and configured LDAP. However, when testing the connection, I received an error message that was not informative at all:
Failed to verify LDAP server with error: error: ldap server network timeout.
Although the error message doesn’t explicitly say there’s a certificate issue and there is nothing in the harbor-core container logs, it immediately made sense to me that the harbor-core container didn’t trust my LDAPS/CA certificate, so I started investigating how the certificate could be injected somehow into Harbor. The Harbor package doesn’t have any input for the LDAPS/CA certificate in its data values file, so I knew I had to create my own YTT overlay.
Have you ever had to deploy a package using kapp-controller from your Harbor private registry?
I recently deployed the Tanzu RabbitMQ package to a TKGm workload cluster in an air-gapped/internet-restricted environment.
Doing so in air-gapped environments requires you to push the packages into Harbor, then have kapp-controller deploy the package from Harbor.
After adding the PackageRepository referencing my Harbor registry, I observed it couldn’t complete reconciling due to a certificate issue.
Continue reading
A recent project I was working on required granting different levels of permissions for several Active Directory service accounts on Harbor registry so that some can only pull images from the registry, and others can also push, etc.
On the Harbor project, I had the following configuration for my users:
The harbor-group-01 group contains an Active Directory user named harbor-user-01 and harbor-group-02 contains harbor-user-02.
From the command line, I was able to log in to Harbor with harbor-user-01: