HashiCorp Consul Service Mesh on Kubernetes Series - Part 1 - Introduction and Setup

Modern cloud-native architectures rely heavily on microservices, and Kubernetes has become the go-to platform for deploying, managing, and scaling these distributed applications. As the number of microservices grows, ensuring secure, reliable, and observable service-to-service communication becomes increasingly complex. This is where service mesh solutions, such as HashiCorp Consul, step in to provide a seamless approach to managing these challenges. In this blog post, we will delve into the integration of HashiCorp Consul Service Mesh with Kubernetes, exploring its architecture, features, and step-by-step deployment guide.

Why Choose HashiCorp Consul Service Mesh?

HashiCorp Consul stands out for its:

Service Discovery: Automatically registers and discovers services, enabling dynamic and resilient service-to-service communication.
Traffic Management: Provides fine-grained control over traffic routing with features like service segmentation, blue-green deployments, and canary releases.
Zero Trust Security: Implements service-to-service encryption (mTLS) and authentication by default.
Observability: Offers detailed telemetry and metrics for understanding service performance and dependencies.
Multi-Cloud and Hybrid Compatibility: Seamlessly bridges on-premises and cloud environments.

These features make Consul a compelling choice for organizations adopting Kubernetes for their microservices architecture.

Consul Architecture on Kubernetes

When deployed on Kubernetes, Consul integrates deeply into the ecosystem using the following components:

Consul Server Cluster: Acts as the control plane, managing service registrations, health checks, and configurations.
Consul Clients: Proxy agents deployed as sidecars alongside application pods to handle communication and security policies.
Connect (Service Mesh): Manages mTLS encryption, authorization, and traffic routing between services.
Helm Chart: Simplifies deployment and configuration of Consul on Kubernetes.

Below is a diagram showcasing Consul’s architecture on Kubernetes for this implementation:

Part 1: Introduction and Setup

Architecture Overview

HashiCorp Consul is a powerful service mesh solution that provides a comprehensive platform for service networking in modern, distributed architectures. As a service mesh, Consul offers essential features for managing microservices:

Service Discovery: Automatically register and discover services across your infrastructure.
Security: Secure service-to-service communication with automatic mTLS encryption.
Traffic Management: Control and optimize service traffic with advanced routing capabilities.
Observability: Gain insights into your service mesh with integrated monitoring and tracing.

This series will guide you through implementing Consul Service Mesh on Kubernetes, demonstrating how to leverage these features effectively in a production environment.

Deploy Infrastructure Helm Charts

Before deploying Consul, set up the necessary infrastructure components to support a production-grade service mesh.

To follow along, you will need to clone my GitHub repository where all the configuration files are stored:

git clone https://github.com/itaytalmi/consul-service-mesh-k8s.git
cd consul-service-mesh-k8s

GP3 Storage Class

Provides a reliable, cost-effective AWS EBS storage layer for Kubernetes workloads.

kubectl apply -f misc/ebs-csi/ebs-csi-storage-classes.yaml

AWS Load Balancer Controller

Automates the creation and management of AWS Elastic Load Balancers for external access to Kubernetes services.

helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
    -n kube-system \
    -f helm-charts/aws-load-balancer-controller/helm-values.yaml

External DNS

Automatically manages DNS records for Kubernetes services, ensuring seamless service discovery.

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns
helm repo update
helm upgrade -i external-dns external-dns/external-dns \
    -n external-dns --create-namespace \
    -f helm-charts/external-dns/helm-values.yaml

Kube Prometheus Stack

Provides monitoring and alerting using Prometheus, Grafana, and AlertManager for visibility into Consul and other services.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade -i kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    -n kube-prometheus-stack --create-namespace \
    -f helm-charts/kube-prometheus-stack/helm-values.yaml

Jaeger

Enables distributed tracing for root cause analysis and performance optimization in microservices.

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
helm upgrade -i jaeger jaegertracing/jaeger \
    -n jaeger --create-namespace \
    -f helm-charts/jaeger/helm-values.yaml

Consul

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm upgrade -i consul hashicorp/consul -n consul --create-namespace -f helm-charts/consul/helm-values.yaml

Install the Consul CLI. Instructions vary based on the operating system and/or package manager used. See this page for specific instructions.

Using brew, you can also install consul CLI using the following steps:

brew tap hashicorp/tap
brew install hashicorp/tap/consul
consul version

Get the Consul bootstrap ACL token, and ensure you can log in to Consul.

# Set your Consul FQDN
export CONSUL_HTTP_ADDR="https://$(kubectl get ingress consul-ui -n consul -o jsonpath='{.spec.rules[0].host}')"
echo "Consul UI address: $CONSUL_HTTP_ADDR"

# Get Consul bootstrap ACL token
export CONSUL_HTTP_TOKEN=$(kubectl get secret -n consul consul-bootstrap-acl-token -o jsonpath='{.data.token}' | base64 -d)
echo "Consul bootstrap ACL token: $CONSUL_HTTP_TOKEN"

# Verify that you can access Consul
consul members

Since we have enabled Consul ACLs, we must create an ACL token for Prometheus to scrape the Consul Server metrics.

Create an ACL policy with agent:read permissions, create a token for the policy, and store it in a Kubernetes secret. Prometheus Operator will use this secret for Consul authentication.

consul acl policy create -name=agent-read -rules=@consul-policy-agent-read.hcl

CONSUL_POLICY_ACL_TOKEN=$(consul acl token create -policy-name agent-read -format json | jq -r '.SecretID')
echo "Consul policy ACL token: $CONSUL_POLICY_ACL_TOKEN"

kubectl create secret generic consul-token-agent-read -n kube-prometheus-stack --from-literal=token="$CONSUL_POLICY_ACL_TOKEN" --dry-run=client -o yaml | kubectl apply -f -

Create the ProxyDefaults resource containing additional Consul configuration for Kubernetes. The following manifest contains the required configuration for access logs and distributed tracing.

kubectl apply -f consul-proxy-defaults-config.yaml

Deploy Sample App

The sample application that will be used for testing is bookinfo. This is the same sample application used in Istio tutorials, but it has been adapted here for use with Consul.

Create the Bookinfo namespace.

kubectl apply -f bookinfo-ns.yaml

Create the required Consul configuration for the Bookinfo application. The following manifest contains the Consul ServiceDefaults and ServiceIntentions resources.

kubectl apply -f bookinfo-consul-config.yaml -n bookinfo

Deploy the sample application.

kubectl apply -f bookinfo-app.yaml -n bookinfo

Confirm all services and pods are correctly defined and running.

kubectl get pods,svc -n bookinfo

Deploy a Kubernetes Gateway.

kubectl apply -f bookinfo-gateway.yaml -n bookinfo

Note: there is an issue preventing the gateway from registering properly and becoming available. The gateway pod fails to initialize. The connect-inject logs show an error: Service is invalid: spec.loadBalancerClass: Invalid value: null. This occurs when the service type is LoadBalancer. As a workaround, either delete the gateway service or change the service type to ClusterIP or NodePort. Note that deleting the service is the preferred method. The gateway should then reconcile successfully.
kubectl delete svc bookinfo-gateway -n bookinfo

Ensure the Gateway is ready.

kubectl get pods,svc -n bookinfo
kubectl get gateway -n bookinfo

Get the gateway address and port from the bookinfo gateway resource:

export INGRESS_HOST=$(kubectl get svc -n bookinfo bookinfo-gateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
export INGRESS_PORT=$(kubectl get svc -n bookinfo bookinfo-gateway -o jsonpath='{.spec.ports[?(@.name=="http")].port}')
export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
echo $GATEWAY_URL

To confirm the application is running, send a request to the application through the gateway using curl from any pod.

kubectl exec -n bookinfo "$(kubectl get pod -n bookinfo -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS "$GATEWAY_URL/productpage" | grep -o "<title>.*</title>"

Example output:

<title>Simple Bookstore App</title>

Wrap-Up

In this blog post, we introduced the key challenges of managing microservices communication in Kubernetes and how HashiCorp Consul Service Mesh addresses these issues. We also explored the architecture of Consul on Kubernetes, setting the stage for a hands-on deployment experience.

With the infrastructure components and sample application deployed, you’re now equipped to begin leveraging Consul’s capabilities for service discovery, traffic management, security, and observability.

Next Steps

In the upcoming posts in this series, we will focus on three main areas: observability, traffic management, and security. Each area will be broken down into detailed subsections to guide you through the implementation and configuration of HashiCorp Consul Service Mesh in Kubernetes.

Observability

In this section, we will cover:

Visualizing the Mesh
- Learn how to use Consul’s tools and integrations to map out your service mesh.
Querying Metrics with Prometheus
- Configure Prometheus to scrape metrics from Consul and analyze service performance.
Visualizing the Metrics with Grafana
- Set up Grafana dashboards to monitor the health and behavior of your service mesh.
Distributed Tracing with Jaeger
- Integrate Jaeger to trace requests across services, identify bottlenecks, and analyze service dependencies.
Logging and Auditing
- Enable logging and audit capabilities to track activity and troubleshoot issues effectively.

Traffic Management

This section will explore:

Request Routing
- Implement fine-grained traffic routing rules to direct requests between services.
Traffic Shifting
- Set up traffic splitting for use cases like blue-green deployments and canary releases.
Request Timeouts
- Configure request timeouts to improve the reliability and responsiveness of services.
Circuit Breaking
- Prevent cascading failures by implementing circuit breakers in your service mesh.

Security

In this section, we will delve into:

mTLS (Mutual TLS)
- Secure service-to-service communication with automatic mTLS encryption.
Access Control
- Define and enforce granular access policies to control service interactions.
Rate Limiting
- Prevent abuse and ensure fair resource distribution with rate limiting policies.

What We Will Do and See in the Series

This series aims to provide you with:

A step-by-step guide for deploying and configuring Consul Service Mesh on Kubernetes.
Practical examples of how to leverage Consul features in production scenarios.
Detailed insights into observability, traffic management, and security features of Consul.
Best practices to make your Kubernetes microservices architecture secure, resilient, and scalable.

By the end of this series, you will have a comprehensive understanding of how to deploy, manage, and enhance your service mesh using HashiCorp Consul on Kubernetes.

Stay tuned for the next post, where we will dive into observability to monitor and visualize your service mesh effectively!