Customizing Elasticsearch indices using Fluent-Bit in TKG

Fluent-Bit is currently the preferred option for log shipping in TKG and is provided out of the box as a Tanzu package that can be easily deployed on each TKG/Kubernetes cluster.

A recent implementation required shipping all Kubernetes logs to Elasticsearch, complying with a specific naming convention for the Elasticsearch indices.

Applying such customizations requires you to utilize the Lua filter. Using the Lua filter, you can modify incoming records by invoking custom scripts to apply your logic when processing the records.

The Fluent-Bit Tanzu package supports specifying additional filters. However, it currently doesn’t let you define Lua scripts. Therefore, I had to customize the Fluent-Bit package and decided to do so using a YTT overlay, which took some research. :)

This tutorial outlines the steps to deploy the Fluent-Bit Tanzu package, integrate it with Elasticsearch, and apply the custom logic I described above.

Prerequisites

To follow along, you will need a running instance of Elasticsearch and Kibana.

If you don’t already have Elasticsearch and Kibana, you can quickly deploy them on Kubernetes using Bitnami’s Helm chart. Refer to the following example for deploying Elasticsearch and Kibana if necessary:

Clone my TKG GitHub repository and cd into vmware-tkg/tkg-packages/fluent-bit-elasticsearch-customization.

git clone https://github.com/itaytalmi/vmware-tkg.git
cd vmware-tkg/tkg-packages/fluent-bit-elasticsearch-customization

Add the Bitnami Helm repository.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Modify elasticsearch-values.yaml and specify your Elasticsearch and Kibana ingress hostnames and TLS certificates. The certificates should represent the hostnames of Elasticsearch and Kibana (e.g., demo-elasticsearch.terasky.demo, demo-kibana.terasky.demo).

You can further customize your values.yaml file if necessary. Refer to Bitnami’s GitHub repository for more information.

Deploy the Helm chart.

# Version-pinning to prevent breaking changes
CHART_VERSION=19.1.6
helm upgrade -i elasticsearch bitnami/elasticsearch -n elasticsearch -f elasticsearch-values.yaml --create-namespace --version "$CHART_VERSION"

Note: unless you have External DNS installed on your cluster, you will also have to create DNS records for Elasticsearch and Kibana.

Ensure the pods are running and the ingress resources are present.

kubectl get pods,ingress -n elasticsearch

Browse to Elasticsearch and Kibana via a web browser (e.g., https://demo-elasticsearch.terasky.demo, https://demo-kibana.terasky.demo) to ensure both are accessible.

Elasticsearch:

Kibana:

Deploy the Fluent-Bit Tanzu Package

Retrieve the latest version available on your cluster for the Fluent-Bit Tanzu package.

PKG_NAME=fluent-bit.tanzu.vmware.com
PKG_VERSIONS=($(tanzu package available list "$PKG_NAME" -n tanzu-package-repo-global -o json | jq -r ".[].version" | sort -t "." -k1,1n -k2,2n -k3,3n))
PKG_VERSION=${PKG_VERSIONS[-1]}
echo "$PKG_VERSION"

For example: 1.7.5+vmware.2-tkg.1

Deploy the Fluent-Bit package.

tanzu package install fluent-bit \
--package-name "$PKG_NAME" \
--version "$PKG_VERSION" \
--namespace tkg-packages \
--create-namespace

Wait for the deployment to complete.

Installing package 'fluent-bit.tanzu.vmware.com'
Creating namespace 'tkg-packages'
Getting package metadata for 'fluent-bit.tanzu.vmware.com'
Creating service account 'fluent-bit-tkg-packages-sa'
Creating cluster admin role 'fluent-bit-tkg-packages-cluster-role'
Creating cluster role binding 'fluent-bit-tkg-packages-cluster-rolebinding'
Creating package resource
Waiting for 'PackageInstall' reconciliation for 'fluent-bit'
'PackageInstall' resource install status: Reconciling
'PackageInstall' resource install status: ReconcileSucceeded
'PackageInstall' resource successfully reconciled
Added installed package 'fluent-bit'

It is now time to apply our custom logic using the overlay-fluent-bit-lua.yaml YTT overlay file, but first, let’s review it, as there are a few things to note here.

The first part of the overlay adds the generate_es_index_name.lua section containing our Lua script to the fluent-bit-config ConfigMap.

The script modifies the incoming records using the generate_es_index function, handles cluster-level and namespace-level records, and returns the modified record. For cluster-level records, es_index will return a value of $CLUSTER_NAME-$DATE and for namespace-level records, es_index will return a value of $CLUSTER_NAME__$NAMESPACE_NAME-$DATE.

It is then referenced in a Fluent-Bit filter. This filter will call the function for every incoming record.

Finally, in the Elasticsearch output, the Logstash_Prefix_Key uses the es_index key of the modified record, which was generated by the function.

Before applying the overlay, modify the overlay-fluent-bit-lua.yaml to reflect your configuration:

Set your Elasticsearch FQDN in the outputs section.
Set your TKG management cluster name for the tkg_instance parameter (e.g., it-tkg-mgmt-cls) and your TKG workload cluster name for the tkg_cluster parameter (e.g., it-tkg-wld-cls-01) in the filters section.

Create the secret containing the overlay.

kubectl apply -f overlay-fluent-bit-lua.yaml -n tkg-packages

Annotate the Fluent-Bit PackageInstall resource with the secret.

kubectl annotate packageinstalls fluent-bit -n tkg-packages \
ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=overlay-fluent-bit-lua

Delete the Fluent-Bit DaemonSet and trigger reconciliation for the Fluent-Bit package to ensure the Fluent-Bit pods run correctly using the new configuration.

kubectl delete ds fluent-bit -n tanzu-system-logging
kctrl app kick -a fluent-bit -n tkg-packages -y

Ensure the package has been reconciled.

You should now see the newly-created indices!

Based on these indices, you can now create a Kibana Data View (formerly known as Index Pattern).

From the left menu, select Data Views > Create data view.

Then click on Create data view once again.

Enter a name that matches your indices for the data view. For example, the Kubernetes cluster name (e.g., it-tkg-wld-cls-01*). You should see that the name matches all indices for the given Kubernetes cluster - cluster-level indices and namespace-level indices.

Then click on Create data view.

Now navigate to Discover from the left menu and enjoy your logs. :)

It is also worth mentioning that all of the above would also work outside of TKG. If you look at the community’s Fluent-Bit Helm chart, you can see that it accepts Lua scripts as part of its values.yaml file.