Build. Run. Repeat.

HashiCorp Consul Service Mesh on Kubernetes Series - Part 3 - Traffic Management

2025-01-01 10 min read Cloud Native Consul HashiCorp Kubernetes Service Mesh

Efficient traffic management is essential for maintaining application reliability, optimizing performance, and implementing advanced deployment strategies in a service mesh. HashiCorp Consul provides powerful traffic management capabilities through service routers, splitters, and resolvers. In this section, we explore request routing, traffic shifting, request timeouts, and circuit breaking.

Request Routing

This section shows you how to route requests dynamically to multiple versions of a microservice.

The Bookinfo sample consists of four separate microservices, each with multiple versions. Three different versions of one of the microservices, reviews, have been deployed and are running concurrently. To illustrate the problem this causes, access the Bookinfo app’s /productpage in a browser and refresh several times.

You’ll notice that sometimes the book review output contains star ratings and other times it does not. This is because, without an explicit default service version to route to, Consul routes requests to all available versions in a round-robin fashion.

Version 1:

Screenshot

Version 2:

Screenshot

Version 3:

Screenshot

The initial goal of this task is to apply rules that route all traffic to v1 (version 1) of the microservices. Later, we will apply a rule to route traffic based on the value of an HTTP request header.

To route to v1 only, you configure route rules that send traffic to default versions for the microservices.

Consul uses service routers, service splitters, and service resolvers to define route rules. Create rules that will route all traffic to v1 on each microservice.

kubectl -n bookinfo apply -f traffic-management/request-routing/services-all-v1.yaml

You have configured Consul to route to the v1 version of the Bookinfo microservices, most importantly the reviews service version 1.

In the Consul UI, if you go to Services > reviews > Routing, you can view the routing topology.

Screenshot

You can easily test the new configuration by once again refreshing the /productpage of the Bookinfo app in your browser. Notice that the reviews part of the page displays no rating stars, no matter how many times you refresh. This is because you configured Consul to route all traffic for the reviews service to reviews:v1, and this version of the service does not access the star ratings service.

You can also control the route based on specific headers. The productpage service adds a custom end-user header to all outbound HTTP requests to the reviews service. In the following example, we will route all traffic from a user named jason to the service reviews:v2.

Remember, reviews:v2 is the version that includes the star ratings feature.

Apply the routing configuration.

kubectl -n bookinfo apply -f traffic-management/request-routing/service-reviews-v2-header-based.yaml

In the Consul UI, you can view the new routing topology.

Screenshot

On the /productpage of the Bookinfo app, log in as user jason. Note: password is not required.

Screenshot

You should now see the star ratings next to each review. This is because you were routed to reviews:v2.

Screenshot

Sign out and log in as another user (pick any name).

Notice the stars are now gone. This is because the traffic is routed to reviews:v1 for all users except jason.

Screenshot

Remove the routing rules we created.

kubectl -n bookinfo delete -f traffic-management/request-routing/service-reviews-v2-header-based.yaml
kubectl -n bookinfo delete -f traffic-management/request-routing/services-all-v1.yaml

Traffic Shifting

This section shows you how to shift traffic from one version of a microservice to another.

A common use case is to migrate traffic gradually from an older version of a microservice to a new one. In Consul, you accomplish this goal by configuring a sequence of routing rules that redirect a percentage of traffic from one destination to another.

In this task, you will use send 50% of traffic to reviews:v1 and 50% to reviews:v3. Then, you will complete the migration by sending 100% of traffic to reviews:v3.

Create routing rules to route all traffic to v1 on each microservice.

kubectl -n bookinfo apply -f traffic-management/traffic-shifting/services-all-v1.yaml

In the Consul UI, if you go to Services > reviews > Routing, you can view the routing topology.

Screenshot

Open the Bookinfo application in your browser. Notice that the reviews part of the page displays no rating stars, no matter how many times you refresh. This is because you configured Consul to route all traffic for the reviews service to reviews:v1, and this version of the service does not access the star ratings service.

Transfer 50% of the traffic from reviews:v1 to reviews:v3.

kubectl -n bookinfo apply -f traffic-management/traffic-shifting/service-reviews-50-v3.yaml

Refresh the /productpage in your browser, and you now see red-colored star ratings approximately 50% of the time. This is because the v3 version of reviews accesses the star ratings service, but the v1 version does not.

In the Consul UI, you can also see that the traffic is evenly distributed between v1 and v3.

Screenshot

Assuming you decide that the reviews:v3 microservice is stable, you can route 100% of the traffic to reviews:v3 by modifying the routing rules.

kubectl -n bookinfo apply -f traffic-management/traffic-shifting/service-reviews-v3.yaml

In the Consul UI, you can view the new routing topology.

Screenshot

Refresh the /productpage several times. Now you will always see book reviews with red-colored star ratings for each review.

Remove the application routing rules we created.

kubectl -n bookinfo delete -f traffic-management/request-routing/services-all-v1.yaml

Request Timeouts

This section shows you how to set up request timeouts in Envoy using Consul.

A timeout for HTTP requests can be specified using the requestTimeout field of the route rule.

This exercise will use a sample application, NGINX as a frontend service, pointing to httpbin as an upstream API service.

Deploy the application.

kubectl apply -f traffic-management/request-timeouts/sample-app-nginx-httpbin.yaml

Ensure pods are up and running.

kubectl get pod -n httpbin

Use port-forwarding to temporarily access the NGINX frontend service for this exercise.

kubectl port-forward deploy/nginx-frontend 8080:80 -n httpbin

Open 127.0.0.1:8080 in your browser. You should see the httpbin page.

Screenshot

Simulate a delayed response from the httpbin service by artificially adding an 8-second delay.

Browse to 127.0.0.1:8080/delay/8. The page will load for 8 seconds, then return a response.

Screenshot

Screenshot

Now add a 2-second request timeout for calls to the httpbin service.

kubectl apply -f traffic-management/request-timeouts/service-httpbin-timeout.yaml

Refresh the 127.0.0.1:8080/delay/8 page in your browser. The request should time out after 2 seconds.

Screenshot

This is because httpbin takes 8 seconds to return a response but is set to time out after 2 seconds.

Remove the resources we created for this exercise.

kubectl delete -f traffic-management/request-timeouts/service-httpbin-timeout.yaml
kubectl delete -f traffic-management/request-timeouts/sample-app-nginx-httpbin.yaml

Circuit Breaking

This section shows you how to configure circuit breaking for connections, requests, and outlier detection.

Circuit breaking is an important pattern for creating resilient microservice applications. Circuit breaking allows you to write applications that limit the impact of failures, latency spikes, and other undesirable effects of network peculiarities.

In this task, you will configure circuit breaking rules and then test the configuration by intentionally “tripping” the circuit breaker.

Create a client to send traffic to the productpage service. The client is a simple load-testing client called Fortio. Fortio lets you control the number of connections, concurrency, and delays for outgoing HTTP calls. You will use this client to “trip” the circuit breaker policies you set in the ServiceDefaults configuration.

kubectl -n bookinfo apply -f fortio-deploy.yaml

Ensure the Fortio pod is running.

kubectl get pod -n bookinfo -l app=fortio

From a different terminal, tail the log of the productpage service.

kubectl logs -l app=productpage  -c productpage -n bookinfo -f | grep '503\|200'

Get the Fortio pod name.

export FORTIO_POD=$(kubectl get pods -n bookinfo -l app=fortio -o 'jsonpath={.items[0].metadata.name}')
echo "$FORTIO_POD"

Log in to the client pod and use the fortio tool to call productpage. Pass in curl to indicate that you just want to make one call.

kubectl -n bookinfo exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio curl -k -H "x-client-trace-id: 1" -quiet http://bookinfo-gateway/productpage

Example output:

<!DOCTYPE html>
<html>
  <head>
    <title>Simple Bookstore App</title>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">

<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="static/bootstrap/css/bootstrap.min.css">

<!-- Optional theme -->
<link rel="stylesheet" href="static/bootstrap/css/bootstrap-theme.min.css">

  </head>
  <body>
....
</html>
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5289
server: envoy
date: Mon, 13 Feb 2023 22:19:57 GMT
x-envoy-upstream-service-time: 1026

You can see the request succeeded. If you look at the productpage logs in the other terminal, you should see entries indicating status code 200.

Example output:

reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 200 178
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 438
INFO:werkzeug:::ffff:127.0.0.1 - - [20/May/2024 18:30:14] "GET /productpage HTTP/1.1" 200 -

Now, it’s time to break something.

Modify the ServiceDefaults of the productpage service to apply circuit breaking settings when calling its upstream services (details and reviews).

kubectl -n bookinfo apply -f traffic-management/circuit-breaking/service-productpage-circuit-breaking.yaml

In the ServiceDefaults settings, you specified maxConnections: 1 and maxPendingRequests: 1. These rules indicate that if you exceed more than one connection and request concurrently, you should see some failures when the consul-dataplane opens the circuit for further requests and connections.

Call the service with two concurrent connections (-c 2) and send 20 requests (-n 20).

kubectl -n bookinfo exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -k -H "x-client-trace-id: 1" -c 3 -qps 0 -n 30 -loglevel Warning http://productpage/productpage

You should now see some HTTP 503 errors in the productpage logs. For example:

DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 437
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:06:53] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 358
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:06:53] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 503 19
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 503 19
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 503 81
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 503 81
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:06:53] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 437
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:06:53] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 503 19
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 437
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:06:53] "GET /productpage HTTP/1.1" 200 -

Bring the number of concurrent connections up to 3.

kubectl -n consul-bookinfo exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -k -H "x-client-trace-id: 1" -c 3 -qps 0 -n 30 -loglevel Warning https://bookinfo-gateway/productpage

You should now see even more 503 errors in the logs.

DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 503 19
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 358
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:07:45] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 503 81
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 503 81
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:07:45] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 200 437
INFO:werkzeug:::ffff:127.0.0.1 - - [15/Feb/2023 19:07:45] "GET /productpage HTTP/1.1" 200 -
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 503 19
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://details:9080 "GET /details/0 HTTP/1.1" 503 19
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 503 81
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
DEBUG:urllib3.connectionpool:http://reviews:9080 "GET /reviews/0 HTTP/1.1" 503 81

You can also view the metrics for these error events on Prometheus. Open the Prometheus UI in your browser, filter for envoy_cluster_outlier_detection_ejections_active in the expression field and click Execute. You should see many entries matching the expression. The entries represent the errors we previously observed in the productpage logs.

Screenshot

You can also click the Graph tab to view the metrics as a graph.

Screenshot

Revert the application configuration to its original state to remove the circuit breaking configuration.

kubectl apply -f consul/traffic-management/circuit-breaking/service-productpage-defaults.yaml

Wrap-Up

Effective traffic management is crucial for deploying and maintaining resilient microservices. By leveraging Consul’s request routing, traffic shifting, timeouts, and circuit breaking capabilities, you can optimize performance, ensure reliability, and support advanced deployment strategies.

In the next section, we will explore Security features, focusing on mTLS, access control, and rate limiting.

Api Configuration Consul Deployment Hashicorp Ingress Kubernetes Logging Networking Prometheus Security Servicemesh Tanzu Terraform Tls Troubleshooting
© 2025 Build. Run. Repeat. by Itay Talmi - Credits
Bilberry Hugo Theme