HashiCorp Vault Enterprise - Performance Replication on Kubernetes

This blog post dives into the technical implementation of Vault Enterprise replication within a Kubernetes environment. We’ll explore how to set up performance and disaster recovery replication, overcome common challenges, and ensure smooth synchronization between clusters. Whether you’re aiming for redundancy or better data locality, this guide will equip you with the insights and tools needed to leverage Vault’s enterprise-grade features in Kubernetes effectively.

Architecture

Prerequisites

• 2 Kubernetes clusters. *Note: for simulation purposes, you can also use a single Kubernetes cluster with multiple namespaces to host both Vault clusters.
• Helm installed
• kubectl installed
• Vault CLI installed
• jq installed
• Vault Enterprise license

Note: for this implementation LoadBalancer services are used on Kubernetes to expose the Vault services (the API/UI and the cluster address for replication). It is highly recommended to use a LoadBalancer rather than ingress to expose the cluster address for replication. Vault itself performs the TLS termination as the TLS certificates are mounted to the Vault pods from Kubernetes. Additionally, note that when enabling the replication, the primary cluster points to the secondary cluster address (port 8201) and not the API/UI address (port 8200). When the secondary cluster applies the replication token, however, it points to the API/UI address (port 8200) to unwrap it and compelete the setup of the replication. We will see this in more detail in the implementation section.

HashiCorp Vault Intermediate CA Setup with Cert-Manager and Microsoft Root CA

In this post, we’ll explore how to set up HashiCorp Vault as an Intermediate Certificate Authority (CA) on a Kubernetes cluster, using a Microsoft CA as the Root CA. We’ll then integrate this setup with cert-manager, a powerful Kubernetes add-on for automating the management and issuance of TLS certificates.

The following is an architecture diagram for the use case I’ve built.

• A Microsoft Windows server is used as the Root CA of the environment.
• A Kubernetes cluster hosting shared/common services, including HashiCorp Vault. This is a cluster that can serve many other purposes/solutions, consumed by other clusters. The Vault server is deployed on this cluster and serves as an intermediate CA server, under the Microsoft Root CA server.
• A second Kubernetes cluster hosting the application(s). Cert-Manager is deployed on this cluster, integrated with Vault, and handles the management and issuance of TLS certificates against Vault using the ClusterIssuer resource. A web application, exposed via ingress, is running on this cluster. The ingress resource consumes its TLS certificate from Vault.

Prerequisites

• Atleast one running Kubernetes cluster. To follow along, you will need two Kubernetes clusters, one serving as the shared services cluster and the other as the workload/application cluster.
• Access to a Microsoft Root Certificate Authority (CA).
• The Helm CLI installed.
• Clone my GitHub repository. This repository contains all involved manifests, files and configurations needed.

Setting Up HashiCorp Vault as Intermediate CA

Deploy Initialize and Configure Vault

Install the Vault CLI. In the following example, Linux Ubuntu is used. If you are using a different operating system, refer to these instructions.

Using HashiCorp Vault as Ingress TLS Certificate Issuer in TAP

Using HashiCorp Vault as Ingress TLS Certificate Issuer in TAP

Tanzu Application Platform (TAP) uses Contour HTTPProxy resources to expose several web components externally via ingress. Some of these components include the API Auto Registration, API Portal, Application Live View, Metadata Store, and TAP GUI. Web workloads deployed through TAP also leverage the same method for their ingress resources. For example:

$ kubectl get httpproxy -A

NAMESPACE               NAME                                                              FQDN                                                     TLS SECRET                                               STATUS   STATUS DESCRIPTION
api-auto-registration   api-auto-registration-controller                                  api-auto-registration.tap.cloudnativeapps.cloud          api-auto-registration-cert                               valid    Valid HTTPProxy
api-portal              api-portal                                                        api-portal.tap.cloudnativeapps.cloud                     api-portal-tls-cert                                      valid    Valid HTTPProxy
app-live-view           appliveview                                                       appliveview.tap.cloudnativeapps.cloud                    appliveview-cert                                         valid    Valid HTTPProxy
metadata-store          amr-cloudevent-handler-ingress                                    amr-cloudevent-handler.tap.cloudnativeapps.cloud         amr-cloudevent-handler-ingress-cert                      valid    Valid HTTPProxy
metadata-store          amr-graphql-ingress                                               amr-graphql.tap.cloudnativeapps.cloud                    amr-ingress-cert                                         valid    Valid HTTPProxy
metadata-store          metadata-store-ingress                                            metadata-store.tap.cloudnativeapps.cloud                 ingress-cert                                             valid    Valid HTTPProxy
tap-demo-01             spring-petclinic-contour-76691bbb1936a7b010ca900ce58a3f57spring   spring-petclinic.tap-demo-01.svc.cluster.local                                                                    valid    Valid HTTPProxy
tap-demo-01             spring-petclinic-contour-88f827fbdc09abbb4ee2b887bba100edspring   spring-petclinic.tap-demo-01.tap.cloudnativeapps.cloud   tap-demo-01/route-a4b7b2c7-0a56-48b9-ad26-6b0e06ca1925   valid    Valid HTTPProxy
tap-demo-01             spring-petclinic-contour-spring-petclinic.tap-demo-01             spring-petclinic.tap-demo-01                                                                                      valid    Valid HTTPProxy
tap-demo-01             spring-petclinic-contour-spring-petclinic.tap-demo-01.svc         spring-petclinic.tap-demo-01.svc                                                                                  valid    Valid HTTPProxy
tap-gui                 tap-gui                                                           tap-gui.tap.cloudnativeapps.cloud                        tap-gui-cert                                             valid    Valid HTTPProxy

TAP uses a shared ingress issuer as a centralized certificate authority representation, providing a method to set up TLS for the entire platform. All participating components get their ingress certificates issued by it. This is the recommended best practice for issuing ingress certificates on the platform.

Kubernetes Data Protection: Getting Started with Kasten (K10)

In a recent Kubernetes project I was involved in, our team had to conduct an in-depth proof of concept for several Kubernetes data protection solutions. The main highlights of the PoC covered data protection for stateful applications and databases, disaster recovery, and application mobility, including relocating applications across Kubernetes clusters and even different types of Kubernetes clusters (for example, from TKG on-premise to AWS EKS, etc.).

One of the solutions we evaluated was Kasten (K10), a data management platform for Kubernetes, which is now a part of Veeam. The implementation of Kasten was one of the smoothest we have ever experienced in terms of ease of use, stability, and general clarity around getting things done, as everything is very well documented, which certainly cannot be taken for granted these days. :)