CAPV: Addressing Node Provisioning Issues Due to an Invalid State of ETCD

I recently ran into a strange scenario on a Kubernetes cluster after a sudden and unexpected crash it had experienced due to an issue in the underlying vSphere environment. In this case, the cluster was a TKG cluster (in fact, it happened to be the TKG management cluster), however, the same situation could have occurred on any cluster managed by Cluster API Provider vSphere (CAPV).

I have seen clusters unexpectedly crash many times before and most of the time, they successfully went back online when all nodes were up and running. In this case, however, some of the nodes could not boot properly, and Cluster API started attempting their reconciliation.

TKG: Updating Pinniped Configuration and Addressing Common Issues

Most of the TKG engagements I’ve been involved in included Pinniped for Kubernetes authentication. On many occasions, I have seen issues where the configuration provided to Pinniped was incorrect or partially incorrect. For example, common issues may be related to the LDAPS integration. Many environments I have seen utilize Active Directory as the authentication source, and Pinniped requires the LDAPS certificate, username, and password, which are often specified incorrectly. Since this configuration is not validated during the deployment, you end up with an invalid state of Pinniped on your management cluster.

Production-Grade Multi-Cluster TAP Installation Guide

• Introduction
• Prerequisites
• Prepare your Workstation
• Relocate TAP Images to your Private Registry
• Install TAP
  • View Cluster
    • Set up the Installation Namespace
    • Issue a TLS Certificate for TAP GUI
    • Set up a Database for TAP GUI
    • Set up the TAP GUI Catalog Git Repository
    • Set up RBAC for the Metadata Store
    • Set up an Authentication Provider for TAP GUI
    • Set up RBAC for the Build, Run and Iterate Clusters
    • Set an Ingress Domain, TAP GUI Hostname and CA Certificate
    • Deploy the TAP Package
  • Build Cluster
    • Set up the Installation Namespace
    • Set up Metadata Store Authentication and CA Certificate
    • Prepare a Sample Source Code Git Repository
    • Update the TAP Values File
    • Deploy the TAP Package
    • Deploy the TBS Full Dependencies Package
    • Set up the Developer Namespace and Deploy a Workload
  • Run Cluster
    • Set up the Installation Namespace
    • Update the TAP Values File
    • Deploy the TAP Package
    • Set up the Developer Namespace and Deploy a Workload
  • Iterate Cluster
    • Set up the Installation Namespace
    • Update the TAP Values File
    • Deploy the TAP Package
    • Deploy the TBS Full Dependencies Package
    • Set up the Developer Namespace and Deploy a Workload
    • Iterate on your Application
• Wrap Up

Introduction

Since my previous posts on TAP Overview and Backstage, I have been diving deeper into TAP, trying to establish the practices around it.