Harbor on Build. Run. Repeat.

Harbor Registry - Automating LDAP/S Configuration - Part 1

Fri, 01 Nov 2024 09:00:00 -0400

The Harbor Registry is involved in many of my Kubernetes implementations in the field, and in almost every implementation I am asked about the options to configure LDAP/S authentication for the registry. Unfortuntely, neither the community Helm chart nor the Tanzu Harbor package provides native inputs for this setup. Fortunately, the Harbor REST API enables LDAP configuration programmatically. Automating this process ensures consistency across environments, faster deployments, and reduced chances of human error.

HashiCorp Vault Intermediate CA Setup with Cert-Manager and Microsoft Root CA

Mon, 01 Jan 2024 09:00:00 -0400

In this post, we’ll explore how to set up HashiCorp Vault as an Intermediate Certificate Authority (CA) on a Kubernetes cluster, using a Microsoft CA as the Root CA. We’ll then integrate this setup with cert-manager, a powerful Kubernetes add-on for automating the management and issuance of TLS certificates.

The following is an architecture diagram for the use case I’ve built.

A Microsoft Windows server is used as the Root CA of the environment.
A Kubernetes cluster hosting shared/common services, including HashiCorp Vault. This is a cluster that can serve many other purposes/solutions, consumed by other clusters. The Vault server is deployed on this cluster and serves as an intermediate CA server, under the Microsoft Root CA server.
A second Kubernetes cluster hosting the application(s). Cert-Manager is deployed on this cluster, integrated with Vault, and handles the management and issuance of TLS certificates against Vault using the ClusterIssuer resource. A web application, exposed via ingress, is running on this cluster. The ingress resource consumes its TLS certificate from Vault.

Prerequisites

Atleast one running Kubernetes cluster. To follow along, you will need two Kubernetes clusters, one serving as the shared services cluster and the other as the workload/application cluster.
Access to a Microsoft Root Certificate Authority (CA).
The Helm CLI installed.
Clone my GitHub repository. This repository contains all involved manifests, files and configurations needed.

Setting Up HashiCorp Vault as Intermediate CA

Deploy Initialize and Configure Vault

Install the Vault CLI. In the following example, Linux Ubuntu is used. If you are using a different operating system, refer to these instructions.

CAPV: Addressing Node Provisioning Issues Due to an Invalid State of ETCD

Fri, 01 Dec 2023 09:00:00 -0400

I recently ran into a strange scenario on a Kubernetes cluster after a sudden and unexpected crash it had experienced due to an issue in the underlying vSphere environment. In this case, the cluster was a TKG cluster (in fact, it happened to be the TKG management cluster), however, the same situation could have occurred on any cluster managed by Cluster API Provider vSphere (CAPV).

I have seen clusters unexpectedly crash many times before and most of the time, they successfully went back online when all nodes were up and running. In this case, however, some of the nodes could not boot properly, and Cluster API started attempting their reconciliation.

TKG 2.3: Fixing the Prometheus Data Source in the Grafana Package

Fri, 01 Sep 2023 09:00:00 -0400

With the release of TKG 2.3, the Grafana package was finally updated from version 7.5.x to 9.5.1. If you have deployed the new Grafana package (9.5.1+vmware.2-tkg.1) or upgraded your existing one to this version, you may have run into error messages in your Grafana dashboards.

For example, in the TKG Kubernetes cluster monitoring default dashboard, you may have run into the Failed to call resource error when opening the dashboard and noticed that a lot of the data is missing.

Streamlining and Customizing Windows Image Builder for TKG

Wed, 01 Mar 2023 09:00:00 -0400

Tanzu Kubernetes Grid (TKG) is one of the few platforms providing out-of-the-box support and streamlined deployment of Windows Kubernetes clusters. VMware is actively investing in this area and constantly improving the support and capabilities around Windows on Kubernetes.

Unlike Linux-based clusters, for which VMware provides pre-packaged base OS images (typically based on Ubuntu and Photon OS), VMware cannot offer Windows pre-packaged images, primarily due to licensing restrictions, I suppose. Therefore, building your own Windows base OS image is one of the prerequisites for deploying a TKG Windows workload cluster. Fortunately, VMware leverages the upstream Image Builder project - a fantastic collection of cross-provider Kubernetes virtual machine image-building utilities intended to simplify and streamline the creation of base OS images for Kubernetes.

Tanzu Kubernetes Grid GPU Integration

Wed, 01 Mar 2023 09:00:00 -0400

I recently had to demonstrate Tanzu Kubernetes Grid and its GPU integration capabilities. Developing a good use case and assembling the demo required some preliminary research.

During my research, I reached out to Jay Vyas, staff engineer at VMware, SIG Windows lead for Kubernetes, a Kubernetes legend, and an awesome guy in general. :) For those who don’t know Jay, he is also one of the authors of the fantastic book Core Kubernetes (look it up!).

Harbor Registry – Automating LDAP/S Configuration – Part 2

Sun, 01 Jan 2023 09:00:00 -0400

This post continues our two-part series on automating LDAP configuration for Harbor Registry. In the previous post, we demonstrated how to achieve this using Ansible, running externally. However, external automation has its challenges, such as firewall restrictions or limited API access in some cases/environments.

Note: make sure you review the previous post as it provides a lot of additional background and clarifications on this process, LDAPS configuration, and more.

Here, we explore an alternative approach using Terraform, running the automation directly inside the Kubernetes cluster hosting Harbor. This method leverages native Kubernetes scheduling capabilities for running the configuration job in a fully declarative approach and does not require any network access to Harbor from the machine running the job.

Upgrading NSX ALB in a TKG Environment

Thu, 01 Sep 2022 09:00:00 -0400

For quite a long time, the highest version of the NSX ALB TKG supported was 20.1.6/20.1.3, although 21.1.x has been available for a while, and I have been wondering when TKG would support it. In the release notes of TKG 1.5.4, I recently noticed a note that has been added regarding NSX ALB 21.1.x under the Configuration variables section:

AVI_CONTROLLER_VERSION sets the NSX Advanced Load Balancer (ALB) version for NSX ALB v21.1.x deployments in Tanzu Kubernetes Grid.

Getting Harbor to trust your LDAPS certificate in TKG

Mon, 01 Aug 2022 09:00:00 -0400

In a recent TKG implementation, it was required to configure Harbor with LDAPS rather than LDAP.

I deployed the Harbor package on the TKG shared services cluster and configured LDAP. However, when testing the connection, I received an error message that was not informative at all:

Failed to verify LDAP server with error: error: ldap server network timeout.

Although the error message doesn’t explicitly say there’s a certificate issue and there is nothing in the harbor-core container logs, it immediately made sense to me that the harbor-core container didn’t trust my LDAPS/CA certificate, so I started investigating how the certificate could be injected somehow into Harbor. The Harbor package doesn’t have any input for the LDAPS/CA certificate in its data values file, so I knew I had to create my own YTT overlay.

Getting kapp-controller to trust your CA certificates in TKG

Mon, 01 Aug 2022 09:00:00 -0400

Have you ever had to deploy a package using kapp-controller from your Harbor private registry?

I recently deployed the Tanzu RabbitMQ package to a TKGm workload cluster in an air-gapped/internet-restricted environment.

Doing so in air-gapped environments requires you to push the packages into Harbor, then have kapp-controller deploy the package from Harbor.

After adding the PackageRepository referencing my Harbor registry, I observed it couldn’t complete reconciling due to a certificate issue.

Harbor Registry: is your LDAP user unique?

Mon, 01 Aug 2022 09:00:00 -0400

A recent project I was working on required granting different levels of permissions for several Active Directory service accounts on Harbor registry so that some can only pull images from the registry, and others can also push, etc.

On the Harbor project, I had the following configuration for my users:

The harbor-group-01 group contains an Active Directory user named harbor-user-01 and harbor-group-02 contains harbor-user-02.

From the command line, I was able to log in to Harbor with harbor-user-01:

Production-Grade Multi-Cluster TAP Installation Guide

Mon, 01 Aug 2022 09:00:00 -0400

Introduction
Prerequisites
Prepare your Workstation
Relocate TAP Images to your Private Registry
Install TAP
Wrap Up

Introduction

Since my previous posts on TAP Overview and Backstage, I have been diving deeper into TAP, trying to establish the practices around it.