Getting kapp-controller to trust your CA certificates in TKG

Have you ever had to deploy a package using kapp-controller from your Harbor private registry?

I recently deployed the Tanzu RabbitMQ package to a TKGm workload cluster in an air-gapped/internet-restricted environment.

Doing so in air-gapped environments requires you to push the packages into Harbor, then have kapp-controller deploy the package from Harbor.

After adding the PackageRepository referencing my Harbor registry, I observed it couldn’t complete reconciling due to a certificate issue.

I realized the reason was that kapp-controller didn’t trust Harbor’s CA certificate. So I had to get kapp-controller to trust my CA using a YTT overlay since a ClusterResourceSet deploys kapp-controller itself, which is rendered/templated by YTT from the TKG management cluster.

I then found and modified the kapp-controller add-on source template under ~/.config/tanzu/tkg/providers/ytt/02_addons/kapp-controller/add_kapp-controller.yaml, and added this overlay.

As you can see, the overlay adds a volume mount to the kapp-controller deployment, referencing a volume that contains the CA certificate. The overlay expects the ca-cert.pem file to be present in the same directory as the overlay so it can read the CA certificate. I placed my PEM-encoded certificate under ~/.config/tanzu/tkg/providers/ytt/02_addons/kapp-controller/ca-cert.pem.

That ensures that kapp-controller will trust the CA certificate on any newly-deployed workload clusters. However, it doesn’t get kapp-controller on existing clusters to trust the CA certificate, so I came up with this shell script for existing clusters.

The script is executed at the TKG management cluster level, loops through all present workload clusters, and patches the kapp-controller add-on secret, injecting the CA certificate overlay.

I then observed my Tanzu RabbitMQ PackageRepository had reconciled successfully, and I could proceed with my deployment.

You can find the above overlay and script, as well as detailed usage instructions and examples, on my TKG repository on GitHub.

I hope it helps anyone looking to deploy packages from a private registry using kapp-controller.