I recently ran into an issue causing the vCenter server to crash almost daily. What seemed to be a random vCenter issue initially, turned out to be related to CAPV (Cluster API Provider vSphere), running on some of our Kubernetes clusters. That was also an edge case I had not seen before, so I decided to document and share it here.
Initially, the issue we were witnessing on the vCenter server was the following:
Continue reading
I recently had to demonstrate Tanzu Kubernetes Grid and its GPU integration capabilities. Developing a good use case and assembling the demo required some preliminary research.
During my research, I reached out to Jay Vyas, staff engineer at VMware, SIG Windows lead for Kubernetes, a Kubernetes legend, and an awesome guy in general. :) For those who don’t know Jay, he is also one of the authors of the fantastic book Core Kubernetes (look it up!).
This post continues our two-part series on automating LDAP configuration for Harbor Registry. In the previous post, we demonstrated how to achieve this using Ansible, running externally. However, external automation has its challenges, such as firewall restrictions or limited API access in some cases/environments.
Note: make sure you review the previous post as it provides a lot of additional background and clarifications on this process, LDAPS configuration, and more.
Here, we explore an alternative approach using Terraform, running the automation directly inside the Kubernetes cluster hosting Harbor. This method leverages native Kubernetes scheduling capabilities for running the configuration job in a fully declarative approach and does not require any network access to Harbor from the machine running the job.
Continue reading
Fluent-Bit is currently the preferred option for log shipping in TKG and is provided out of the box as a Tanzu package that can be easily deployed on each TKG/Kubernetes cluster.
A recent implementation required shipping all Kubernetes logs to Elasticsearch, complying with a specific naming convention for the Elasticsearch indices.
Applying such customizations requires you to utilize the Lua filter. Using the Lua filter, you can modify incoming records by invoking custom scripts to apply your logic when processing the records.
Continue reading
In a recent TKG implementation, it was required to configure Harbor with LDAPS rather than LDAP.
I deployed the Harbor package on the TKG shared services cluster and configured LDAP. However, when testing the connection, I received an error message that was not informative at all:
Failed to verify LDAP server with error: error: ldap server network timeout.
Although the error message doesn’t explicitly say there’s a certificate issue and there is nothing in the harbor-core container logs, it immediately made sense to me that the harbor-core container didn’t trust my LDAPS/CA certificate, so I started investigating how the certificate could be injected somehow into Harbor. The Harbor package doesn’t have any input for the LDAPS/CA certificate in its data values file, so I knew I had to create my own YTT overlay.
A recent project I was working on required granting different levels of permissions for several Active Directory service accounts on Harbor registry so that some can only pull images from the registry, and others can also push, etc.
On the Harbor project, I had the following configuration for my users:
The harbor-group-01 group contains an Active Directory user named harbor-user-01 and harbor-group-02 contains harbor-user-02.
From the command line, I was able to log in to Harbor with harbor-user-01:
Recently, when working on a TKGm implementation project, I initially ran into an issue that seemed very odd, as I hadn’t encountered such behavior in any other implementation before.
The issue was that a workload cluster deployment hung after deploying the first control plane node. Until then, everything seemed just fine; as the cluster deployment had successfully initialized, NSX ALB had successfully allocated a control plane VIP. After that, however, the deployment had completely hung and seemed like it wouldn’t proceed.
Continue reading