Build. Run. Repeat.

HashiCorp Consul Service Mesh on Kubernetes Series - Part 1 - Introduction and Setup

Wed, 01 Jan 2025 09:00:00 -0400

Modern cloud-native architectures rely heavily on microservices, and Kubernetes has become the go-to platform for deploying, managing, and scaling these distributed applications. As the number of microservices grows, ensuring secure, reliable, and observable service-to-service communication becomes increasingly complex. This is where service mesh solutions, such as HashiCorp Consul, step in to provide a seamless approach to managing these challenges. In this blog post, we will delve into the integration of HashiCorp Consul Service Mesh with Kubernetes, exploring its architecture, features, and step-by-step deployment guide.

HashiCorp Consul Service Mesh on Kubernetes Series - Part 2 - Observability

Wed, 01 Jan 2025 09:00:00 -0400

Modern service meshes require robust observability to ensure seamless operations, proactive troubleshooting, and performance optimization. In this section, we explore the observability features of HashiCorp Consul Service Mesh, including visualizing the service mesh, querying metrics, distributed tracing, and logging and auditing.

Visualizing the Service Mesh

The Consul UI is used for visualizing the service mesh and its topology.

Use the watch command to send requests to the application continually. Make sure HTTP status code 200 is returned in the output.

HashiCorp Consul Service Mesh on Kubernetes Series - Part 3 - Traffic Management

Wed, 01 Jan 2025 09:00:00 -0400

Efficient traffic management is essential for maintaining application reliability, optimizing performance, and implementing advanced deployment strategies in a service mesh. HashiCorp Consul provides powerful traffic management capabilities through service routers, splitters, and resolvers. In this section, we explore request routing, traffic shifting, request timeouts, and circuit breaking.

Request Routing

This section shows you how to route requests dynamically to multiple versions of a microservice.

The Bookinfo sample consists of four separate microservices, each with multiple versions. Three different versions of one of the microservices, reviews, have been deployed and are running concurrently. To illustrate the problem this causes, access the Bookinfo app’s /productpage in a browser and refresh several times.

HashiCorp Consul Service Mesh on Kubernetes Series - Part 4 - Security

Wed, 01 Jan 2025 09:00:00 -0400

Security is a fundamental aspect of any service mesh, ensuring that all service-to-service communication is secure, controlled, and auditable. HashiCorp Consul provides robust security features, including mutual TLS (mTLS), access control, and rate limiting.

mTLS

In this section, we will demonstrate mTLS with Consul. Consul enables and strictly enforces mTLS by default. All traffic sent through the Consul Connect Service Mesh is encrypted.

This section is slightly different from the Istio mTLS section because:

HashiCorp Vault Enterprise - Performance Replication on Kubernetes

Wed, 01 Jan 2025 09:00:00 -0400

This blog post dives into the technical implementation of Vault Enterprise replication within a Kubernetes environment. We’ll explore how to set up performance and disaster recovery replication, overcome common challenges, and ensure smooth synchronization between clusters. Whether you’re aiming for redundancy or better data locality, this guide will equip you with the insights and tools needed to leverage Vault’s enterprise-grade features in Kubernetes effectively.

Architecture

Prerequisites

2 Kubernetes clusters. *Note: for simulation purposes, you can also use a single Kubernetes cluster with multiple namespaces to host both Vault clusters.
Helm installed
kubectl installed
Vault CLI installed
jq installed
Vault Enterprise license

Note: for this implementation LoadBalancer services are used on Kubernetes to expose the Vault services (the API/UI and the cluster address for replication). It is highly recommended to use a LoadBalancer rather than ingress to expose the cluster address for replication. Vault itself performs the TLS termination as the TLS certificates are mounted to the Vault pods from Kubernetes. Additionally, note that when enabling the replication, the primary cluster points to the secondary cluster address (port 8201) and not the API/UI address (port 8200). When the secondary cluster applies the replication token, however, it points to the API/UI address (port 8200) to unwrap it and compelete the setup of the replication. We will see this in more detail in the implementation section.

Harbor Registry - Automating LDAP/S Configuration - Part 1

Fri, 01 Nov 2024 09:00:00 -0400

The Harbor Registry is involved in many of my Kubernetes implementations in the field, and in almost every implementation I am asked about the options to configure LDAP/S authentication for the registry. Unfortuntely, neither the community Helm chart nor the Tanzu Harbor package provides native inputs for this setup. Fortunately, the Harbor REST API enables LDAP configuration programmatically. Automating this process ensures consistency across environments, faster deployments, and reduced chances of human error.

MinIO on vSphere - Automated Deployment and Onboarding

Fri, 01 Nov 2024 09:00:00 -0400

In the world of Kubernetes, reliable S3-compliant object storage is essential for tasks like storing backups. However, not everyone has access to a native S3-compatible solution, and setting one up can feel like a daunting task. MinIO, an open-source object storage solution, is a popular choice to fill this gap. Its lightweight, high-performance architecture makes it an excellent option for Kubernetes users seeking quick and reliable storage.

MinIO is also one of the most widely adopted open-source object storage solutions, thanks to its simplicity and S3 compatibility. It’s perfect for Kubernetes environments that need a reliable and scalable storage layer for backups, logs, or other data.

Fixing Missing TKRs in Existing TKGS Deployments

Wed, 01 May 2024 09:00:00 -0400

I regularly check the Tanzu Kubernetes Releases (TKR) release notes page for new updates. Yesterday, a new TKR was released with support for Kubernetes 1.28.8, and while attempting to test this new version in my TKGS environment, I realized that the TKR was not present in my environment and I started wondering why, as normally, when new TKRs are released, they immediately become available for deployment, since the vCenter is subscribed to the VMware public content library where all the TKRs are hosted. This time, that was not the case, so I started investigating.

HashiCorp Vault Intermediate CA Setup with Cert-Manager and Microsoft Root CA

Mon, 01 Jan 2024 09:00:00 -0400

In this post, we’ll explore how to set up HashiCorp Vault as an Intermediate Certificate Authority (CA) on a Kubernetes cluster, using a Microsoft CA as the Root CA. We’ll then integrate this setup with cert-manager, a powerful Kubernetes add-on for automating the management and issuance of TLS certificates.

The following is an architecture diagram for the use case I’ve built.

A Microsoft Windows server is used as the Root CA of the environment.
A Kubernetes cluster hosting shared/common services, including HashiCorp Vault. This is a cluster that can serve many other purposes/solutions, consumed by other clusters. The Vault server is deployed on this cluster and serves as an intermediate CA server, under the Microsoft Root CA server.
A second Kubernetes cluster hosting the application(s). Cert-Manager is deployed on this cluster, integrated with Vault, and handles the management and issuance of TLS certificates against Vault using the ClusterIssuer resource. A web application, exposed via ingress, is running on this cluster. The ingress resource consumes its TLS certificate from Vault.

Prerequisites

Atleast one running Kubernetes cluster. To follow along, you will need two Kubernetes clusters, one serving as the shared services cluster and the other as the workload/application cluster.
Access to a Microsoft Root Certificate Authority (CA).
The Helm CLI installed.
Clone my GitHub repository. This repository contains all involved manifests, files and configurations needed.

Setting Up HashiCorp Vault as Intermediate CA

Deploy Initialize and Configure Vault

Install the Vault CLI. In the following example, Linux Ubuntu is used. If you are using a different operating system, refer to these instructions.

Using HashiCorp Vault as Ingress TLS Certificate Issuer in TAP

Mon, 01 Jan 2024 09:00:00 -0400

Using HashiCorp Vault as Ingress TLS Certificate Issuer in TAP

Tanzu Application Platform (TAP) uses Contour HTTPProxy resources to expose several web components externally via ingress. Some of these components include the API Auto Registration, API Portal, Application Live View, Metadata Store, and TAP GUI. Web workloads deployed through TAP also leverage the same method for their ingress resources. For example:

$ kubectl get httpproxy -A

NAMESPACE NAME FQDN TLS SECRET STATUS STATUS DESCRIPTION
api-auto-registration api-auto-registration-controller api-auto-registration.tap.cloudnativeapps.cloud api-auto-registration-cert valid Valid HTTPProxy
api-portal api-portal api-portal.tap.cloudnativeapps.cloud api-portal-tls-cert valid Valid HTTPProxy
app-live-view appliveview appliveview.tap.cloudnativeapps.cloud appliveview-cert valid Valid HTTPProxy
metadata-store amr-cloudevent-handler-ingress amr-cloudevent-handler.tap.cloudnativeapps.cloud amr-cloudevent-handler-ingress-cert valid Valid HTTPProxy
metadata-store amr-graphql-ingress amr-graphql.tap.cloudnativeapps.cloud amr-ingress-cert valid Valid HTTPProxy
metadata-store metadata-store-ingress metadata-store.tap.cloudnativeapps.cloud ingress-cert valid Valid HTTPProxy
tap-demo-01 spring-petclinic-contour-76691bbb1936a7b010ca900ce58a3f57spring spring-petclinic.tap-demo-01.svc.cluster.local valid Valid HTTPProxy
tap-demo-01 spring-petclinic-contour-88f827fbdc09abbb4ee2b887bba100edspring spring-petclinic.tap-demo-01.tap.cloudnativeapps.cloud tap-demo-01/route-a4b7b2c7-0a56-48b9-ad26-6b0e06ca1925 valid Valid HTTPProxy
tap-demo-01 spring-petclinic-contour-spring-petclinic.tap-demo-01 spring-petclinic.tap-demo-01 valid Valid HTTPProxy
tap-demo-01 spring-petclinic-contour-spring-petclinic.tap-demo-01.svc spring-petclinic.tap-demo-01.svc valid Valid HTTPProxy
tap-gui tap-gui tap-gui.tap.cloudnativeapps.cloud tap-gui-cert valid Valid HTTPProxy

TAP uses a shared ingress issuer as a centralized certificate authority representation, providing a method to set up TLS for the entire platform. All participating components get their ingress certificates issued by it. This is the recommended best practice for issuing ingress certificates on the platform.

CAPV: Addressing Node Provisioning Issues Due to an Invalid State of ETCD

Fri, 01 Dec 2023 09:00:00 -0400

I recently ran into a strange scenario on a Kubernetes cluster after a sudden and unexpected crash it had experienced due to an issue in the underlying vSphere environment. In this case, the cluster was a TKG cluster (in fact, it happened to be the TKG management cluster), however, the same situation could have occurred on any cluster managed by Cluster API Provider vSphere (CAPV).

I have seen clusters unexpectedly crash many times before and most of the time, they successfully went back online when all nodes were up and running. In this case, however, some of the nodes could not boot properly, and Cluster API started attempting their reconciliation.

CAPV: Fixing and Cleaning Up Idle vCenter Server Sessions

Wed, 01 Nov 2023 09:00:00 -0400

I recently ran into an issue causing the vCenter server to crash almost daily. What seemed to be a random vCenter issue initially, turned out to be related to CAPV (Cluster API Provider vSphere), running on some of our Kubernetes clusters. That was also an edge case I had not seen before, so I decided to document and share it here.

Initially, the issue we were witnessing on the vCenter server was the following:

TKG 2.3: Fixing the Prometheus Data Source in the Grafana Package

Fri, 01 Sep 2023 09:00:00 -0400

With the release of TKG 2.3, the Grafana package was finally updated from version 7.5.x to 9.5.1. If you have deployed the new Grafana package (9.5.1+vmware.2-tkg.1) or upgraded your existing one to this version, you may have run into error messages in your Grafana dashboards.

For example, in the TKG Kubernetes cluster monitoring default dashboard, you may have run into the Failed to call resource error when opening the dashboard and noticed that a lot of the data is missing.

TKG: Updating Pinniped Configuration and Addressing Common Issues

Thu, 01 Jun 2023 09:00:00 -0400

Most of the TKG engagements I’ve been involved in included Pinniped for Kubernetes authentication. On many occasions, I have seen issues where the configuration provided to Pinniped was incorrect or partially incorrect. For example, common issues may be related to the LDAPS integration. Many environments I have seen utilize Active Directory as the authentication source, and Pinniped requires the LDAPS certificate, username, and password, which are often specified incorrectly. Since this configuration is not validated during the deployment, you end up with an invalid state of Pinniped on your management cluster.

Streamlining and Customizing Windows Image Builder for TKG

Wed, 01 Mar 2023 09:00:00 -0400

Tanzu Kubernetes Grid (TKG) is one of the few platforms providing out-of-the-box support and streamlined deployment of Windows Kubernetes clusters. VMware is actively investing in this area and constantly improving the support and capabilities around Windows on Kubernetes.

Unlike Linux-based clusters, for which VMware provides pre-packaged base OS images (typically based on Ubuntu and Photon OS), VMware cannot offer Windows pre-packaged images, primarily due to licensing restrictions, I suppose. Therefore, building your own Windows base OS image is one of the prerequisites for deploying a TKG Windows workload cluster. Fortunately, VMware leverages the upstream Image Builder project - a fantastic collection of cross-provider Kubernetes virtual machine image-building utilities intended to simplify and streamline the creation of base OS images for Kubernetes.

Tanzu Kubernetes Grid GPU Integration

Wed, 01 Mar 2023 09:00:00 -0400

I recently had to demonstrate Tanzu Kubernetes Grid and its GPU integration capabilities. Developing a good use case and assembling the demo required some preliminary research.

During my research, I reached out to Jay Vyas, staff engineer at VMware, SIG Windows lead for Kubernetes, a Kubernetes legend, and an awesome guy in general. :) For those who don’t know Jay, he is also one of the authors of the fantastic book Core Kubernetes (look it up!).

Backstage Introduction, KubeCon & CloudNativeCon Europe 2022

Sun, 01 Jan 2023 09:00:00 -0400

Thanks to TeraSky’s education program, I recently attended KubeCon & CloudNativeCon Europe 2022 in Valencia, Spain.

The experience was incredible. While there were many interesting technical sessions on many exciting topics, I was most curious about Backstage - which has sparked my interest ever since I started exploring VMware Tanzu Application Platform (TAP).

I decided to attend a session entitled “Backstage: Restoring Order to Your Chaos”, given by Spotify software engineer Dave Zolotusky. Going into the session, I was stunned by the huge line of people trying to get into the room. That was something I had never seen before in any other session. Fortunately, I was lucky enough to secure one of the last seats.

Getting Started with Carvel ytt - Real-World Examples

Sun, 01 Jan 2023 09:00:00 -0400

Over the years of working with Tanzu Kubernetes Grid (TKG), one tool has stood out as a game-changer for resource customization: Carvel’s ytt. Whether tailoring cluster manifests, customizing TKG packages, or addressing unique deployment requirements, ytt has consistently been a fundamental part of the workflow. Its flexibility, power, and declarative approach make it an essential tool for anyone working deeply with Kubernetes in a TKG ecosystem.

But what exactly is ytt? Short for YAML Templating Tool, ytt is part of the Carvel suite of tools designed for Kubernetes resource management. It provides a powerful, programmable approach to templating YAML configurations by combining straightforward data values, overlays, and scripting capabilities. Unlike many traditional templating tools, ytt prioritizes structure and intent, making it easier to maintain, validate, and debug configurations—particularly in complex, large-scale Kubernetes environments.

Harbor Registry – Automating LDAP/S Configuration – Part 2

Sun, 01 Jan 2023 09:00:00 -0400

This post continues our two-part series on automating LDAP configuration for Harbor Registry. In the previous post, we demonstrated how to achieve this using Ansible, running externally. However, external automation has its challenges, such as firewall restrictions or limited API access in some cases/environments.

Note: make sure you review the previous post as it provides a lot of additional background and clarifications on this process, LDAPS configuration, and more.

Here, we explore an alternative approach using Terraform, running the automation directly inside the Kubernetes cluster hosting Harbor. This method leverages native Kubernetes scheduling capabilities for running the configuration job in a fully declarative approach and does not require any network access to Harbor from the machine running the job.

Replacing your vCenter server certificate? TKG needs to know about it…

Sun, 01 Jan 2023 09:00:00 -0400

I recently ran into an issue where TKGm had suddenly failed to connect to the vCenter server.

The issue turned out to be TLS-related, and I noticed that the vCenter server certificate had been replaced…

Due to the certificate issue, Cluster API components failed to communicate with vSphere, causing cluster reconciliation to fail, among other vSphere-related operations.

Since all TKG clusters in the environment were deployed with the VSPHERE_TLS_THUMBPRINT parameter specified, replacing the vCenter certificate breaks the connection to vSphere, as the TLS thumbprint changes as well.

Upgrading NSX ALB in a TKG Environment

Thu, 01 Sep 2022 09:00:00 -0400

For quite a long time, the highest version of the NSX ALB TKG supported was 20.1.6/20.1.3, although 21.1.x has been available for a while, and I have been wondering when TKG would support it. In the release notes of TKG 1.5.4, I recently noticed a note that has been added regarding NSX ALB 21.1.x under the Configuration variables section:

AVI_CONTROLLER_VERSION sets the NSX Advanced Load Balancer (ALB) version for NSX ALB v21.1.x deployments in Tanzu Kubernetes Grid.

Customizing Elasticsearch indices using Fluent-Bit in TKG