HashiCorp Consul Service Mesh on Kubernetes Series - Part 4 - Security

Security is a fundamental aspect of any service mesh, ensuring that all service-to-service communication is secure, controlled, and auditable. HashiCorp Consul provides robust security features, including mutual TLS (mTLS), access control, and rate limiting.

mTLS

In this section, we will demonstrate mTLS with Consul. Consul enables and strictly enforces mTLS by default. All traffic sent through the Consul Connect Service Mesh is encrypted.

This section is slightly different from the Istio mTLS section because:

mTLS is strictly enforced by default.
We are using ACLs/intentions (which are not available in Istio).
We are using an ingress gateway to communicate with mesh.

We will now send some traffic to the mesh from both Fortio instances - the one in the bookinfo namespace, where Consul Connect sidecar injection is enabled, and the one in the default namespace, where Consul Connect sidecar injection is not enabled.

Open a new terminal to send the traffic.

export FORTIO_POD_DEFAULT_NS=$(kubectl get pods -l app=fortio -o jsonpath='{.items[0].metadata.name}')
echo "Fortio pod in default namespace: $FORTIO_POD_DEFAULT_NS"

export FORTIO_POD_BOOKINFO_NS=$(kubectl get pods -n consul-bookinfo -l app=fortio -o jsonpath='{.items[0].metadata.name}')
echo "Fortio pod in bookinfo namespace: $FORTIO_POD_BOOKINFO_NS"

watch -n 1 \
"kubectl exec $FORTIO_POD_DEFAULT_NS -c fortio -- /usr/bin/fortio load -c 3 -k -qps 0 -n 5 -loglevel Warning https://bookinfo-gateway.consul-bookinfo/productpage && \
kubectl exec -n consul-bookinfo $FORTIO_POD_BOOKINFO_NS -c fortio -- /usr/bin/fortio load -c 3 -k -qps 0 -n 5 -loglevel Warning https://bookinfo-gateway/productpage"

Inspect the traffic and ensure it’s encrypted using tcpdump. To do so, you can inject a debug container to the productpage pod using the tcpdump image, then analyze the incoming traffic from the Fortio instances.

Add the debug tcpdump container to the productpage pod and inspect the incoming traffic from the API gateway.

kubectl debug -n consul-bookinfo -it $(kubectl get pod -n consul-bookinfo -l app=productpage -o jsonpath='{.items[0].metadata.name}') \
--image=384617649113.dkr.ecr.us-west-2.amazonaws.com/it-service-mesh/tcpdump:latest --target productpage -- tcpdump -A -i any -s0 host bookinfo-gateway.service.dc1.consul

Example output:

19:38:33.042021 eth0  In  IP bookinfo-gateway.service.dc1.consul.32942 > productpage-v1-5885bd6bd-fd7nv.20000: Flags [P.], seq 22839:23078, ack 242364, win 443, options [nop,nop,TS val 1027143094 ecr 1292074613], length 239
E..#.x@...........?...N .....r.+....d......
=8..M.~u.................xe.....26.Kh5.\...|...Z......B...|...4?.
W..l2.y..w.3.^...E6..'.u.....*..A.7.y./.m.A...n..o.E.$.$...q.......W.n...HQ#D   ...'.'.c.apK.....3.TX.w...N....f.u...F.e<....AY.....~....].L.xd|.D.....9M...wx...f.....=C...Kf.&#.u/.I.
19:38:33.042033 eth0  Out IP productpage-v1-5885bd6bd-fd7nv.20000 > bookinfo-gateway.service.dc1.consul.32942: Flags [.], ack 23078, win 442, options [nop,nop,TS val 1292074616 ecr 1027143094], length 0
E..4.|@.......?.....N ...r.+...............
M.~x=8..

As you can see, the data is unreadable, as it is encrypted.

Access Control

This section shows you how to control access between services using Consul application-aware intentions (layer 7 access).

Open the Bookinfo web application in your browser.

On the product page, you can see the following sections:

Book Details on the lower left side, which includes: book type, number of pages, publisher, etc.
Book Reviews on the lower right of the page.

When you refresh the page, the app shows different versions of reviews on the product page. The app presents the reviews in a round robin style: red stars, black stars, or no stars.

Using Consul, you can easily setup access control for workloads in your mesh. This task shows you how to set up access control using Consul ServiceIntentions. First, you configure a simple deny-all policy that rejects all requests to the workload, and then grant more access to the workload gradually and incrementally.

For this exercise, delete the existing ServiceIntentions from the consul-bookinfo namespace.

kubectl delete serviceintentions.consul.hashicorp.com --all -n consul-bookinfo

Create the deny-all intention to block all traffic and deny all requests.

kubectl apply -f consul/security/access-control/service-intention-deny-all.yaml

Refresh Bookinfo web application in your browser.

The /productpage page loads successfully because the API gateway has access to it using an HTTPRoute. However, you can see the following errors on the page:

Error fetching product details!
Error fetching product reviews

However, the productpage service is unable to reach the details and the reviews service because we have not configured Consul to allow this traffic, and the deny-all rule blocks it.

Create the details-viewer and the reviews-viewer intention to allow GET requests from the productpage workload.

kubectl apply -f consul/security/access-control/service-intention-details-viewer.yaml
kubectl apply -f consul/security/access-control/service-intention-reviews-viewer.yaml

Refresh the Bookinfo application in your browser.

Now, you should see the Bookinfo Sample page with Book Details on the lower left part, and Book Reviews on the lower right part. However, in the Book Reviews section, there is an error Ratings service currently unavailable. This is because the reviews workload doesn’t have permission to access the ratings workload. To fix this issue, you need to grant the reviews workload access to the ratings workload.

Next, we configure an intention to grant the reviews workload access to ratings.

kubectl apply -f consul/security/access-control/service-intention-ratings-viewer.yaml

Refresh the Bookinfo application in your browser. You should see the black and red ratings in the Book Reviews section.

It is also worth noting that since we are using layer 7 application-aware intentions, the Consul UI also displays them differently. For example:

Remove the intentions we created in this section and re-apply the original ones.

kubectl delete -f consul/security/access-control/service-intention-details-viewer.yaml
kubectl delete -f consul/security/access-control/service-intention-reviews-viewer.yaml
kubectl delete -f consul/security/access-control/service-intention-ratings-viewer.yaml
kubectl delete -f consul/security/access-control/service-intention-deny-all.yaml
kubectl apply -f consul/security/access-control/service-intentions.yaml

Rate Limiting

The rate limiting feature support is currently limited in Consul. Consul allows you to configure the maximum inbound connections for a service. However, it currently doesn’t allow you to scope the rate limits to a local/global level, as opposed to Istio. Additionally, setting intervals is currently not possible. Therefore, we will demonstrate a basic example.

Open a new terminal and send traffic to the mesh using Fortio.

export FORTIO_POD=$(kubectl get pods -n consul-bookinfo -l app=fortio -o 'jsonpath={.items[0].metadata.name}')
echo "$FORTIO_POD"

watch -n 1 \
"kubectl exec -n consul-bookinfo $FORTIO_POD -c fortio -- /usr/bin/fortio load -k -c 2 -qps 0 -n 2 -loglevel Warning https://bookinfo-gateway/productpage"

All requests should go through and you should get HTTP status code 200.

Example output:

20:28:08 I logger.go:148> Log level is now 3 Warning (was 2 Info)
Fortio 1.39.1 running at 0 queries per second, 4->4 procs, for 30 calls: https://bookinfo-gateway/productpage
Starting at max qps with 3 thread(s) [gomax 4] for exactly 30 calls (10 per thread + 0)
Ended after 691.998863ms : 30 calls. qps=43.353
Aggregated Function Time : count 30 avg 0.063823179 +/- 0.01996 min 0.032082364 max 0.097597786 sum 1.91469537
# range, mid point, percentile, count
>= 0.0320824 <= 0.035 , 0.0335412 , 10.00, 3
> 0.035 <= 0.04 , 0.0375 , 13.33, 1
> 0.04 <= 0.045 , 0.0425 , 26.67, 4
> 0.045 <= 0.05 , 0.0475 , 36.67, 3
> 0.05 <= 0.06 , 0.055 , 43.33, 2
> 0.06 <= 0.07 , 0.065 , 50.00, 2
> 0.07 <= 0.08 , 0.075 , 76.67, 8
> 0.08 <= 0.09 , 0.085 , 90.00, 4
> 0.09 <= 0.0975978 , 0.0937989 , 100.00, 3
# target 50% 0.07
# target 75% 0.079375
# target 90% 0.09
# target 99% 0.096838
# target 99.9% 0.0975218
Error cases : no data
# Socket and IP used for each connection:
[0]   1 socket used, resolved to 10.100.172.224:443, connection timing : count 1 avg 0.013811518 +/- 0 min 0.013811518 max 0.013811518 sum 0.013811518
[1]   1 socket used, resolved to 10.100.172.224:443, connection timing : count 1 avg 0.010432793 +/- 0 min 0.010432793 max 0.010432793 sum 0.010432793
[2]   1 socket used, resolved to 10.100.172.224:443, connection timing : count 1 avg 0.006944071 +/- 0 min 0.006944071 max 0.006944071 sum 0.006944071
Connection time (s) : count 3 avg 0.010396127 +/- 0.002804 min 0.006944071 max 0.013811518 sum 0.031188382
Sockets used: 3 (for perfect keepalive, would be 3)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
10.100.172.224:443: 3
Code 200 : 30 (100.0 %)
Response Header Sizes : count 30 avg 168 +/- 0 min 168 max 168 sum 5040
Response Body/Total Sizes : count 30 avg 5127 +/- 470.2 min 4462 max 5462 sum 153810
All done 30 calls (plus 0 warmup) 63.823 ms avg, 43.4 qps

Note the Code 200 : 2 (100.0 %) part.

Now apply the maxInboundConnections configuration. We are limiting the number of inbound connections to 1.

kubectl apply -f consul/security/rate-limiting/service-productpage-max-inbound-connections.yaml

You should now start seeing errors.

Example output:

20:33:11 I logger.go:148> Log level is now 3 Warning (was 2 Info)
Fortio 1.39.1 running at 0 queries per second, 4->4 procs, for 2 calls: https://bookinfo-gateway/productpage
Starting at max qps with 2 thread(s) [gomax 4] for exactly 2 calls (1 per thread + 0)
20:33:11 W http_client.go:956> [0] Non ok http code 503 (HTTP/1.1 503)
20:33:11 W http_client.go:956> [1] Non ok http code 503 (HTTP/1.1 503)
Ended after 10.051919ms : 2 calls. qps=198.97
Aggregated Function Time : count 2 avg 0.009848738 +/- 5.651e-05 min 0.009792226 max 0.00990525 sum 0.019697476
# range, mid point, percentile, count
>= 0.00979223 <= 0.00990525 , 0.00984874 , 100.00, 2
# target 50% 0.00979223
# target 75% 0.00984874
# target 90% 0.00988265
# target 99% 0.00990299
# target 99.9% 0.00990502
Error cases : count 2 avg 0.009848738 +/- 5.651e-05 min 0.009792226 max 0.00990525 sum 0.019697476
# range, mid point, percentile, count
>= 0.00979223 <= 0.00990525 , 0.00984874 , 100.00, 2
# target 50% 0.00979223
# target 75% 0.00984874
# target 90% 0.00988265
# target 99% 0.00990299
# target 99.9% 0.00990502
# Socket and IP used for each connection:
[0]   1 socket used, resolved to 10.100.172.224:443, connection timing : count 1 avg 0.005277078 +/- 0 min 0.005277078 max 0.005277078 sum 0.005277078
[1]   1 socket used, resolved to 10.100.172.224:443, connection timing : count 1 avg 0.0047511 +/- 0 min 0.0047511 max 0.0047511 sum 0.0047511
Connection time (s) : count 2 avg 0.005014089 +/- 0.000263 min 0.0047511 max 0.005277078 sum 0.010028178
Sockets used: 2 (for perfect keepalive, would be 2)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
10.100.172.224:443: 2
Code 503 : 2 (100.0 %)
Response Header Sizes : count 2 avg 0 +/- 0 min 0 max 0 sum 0
Response Body/Total Sizes : count 2 avg 225 +/- 0 min 225 max 225 sum 450
All done 2 calls (plus 0 warmup) 9.849 ms avg, 199.0 qps

Note the Code 503 : 2 (100.0 %) part.

This is because we are sending many requests simultaneously and exceeding the inbound limit (1).

Revert the application configuration to its original state to remove the inbound connection limit.

kubectl apply -f consul/traffic-management/circuit-breaking/service-productpage-defaults.yaml

Series Wrap-Up

Throughout this series, we explored the powerful features of HashiCorp Consul Service Mesh on Kubernetes. From observability tools like Prometheus, Grafana, and Jaeger, to advanced traffic management strategies and robust security features, Consul enables seamless management of service-to-service communication in microservices architectures.

Key takeaways:

Observability: Visualize, monitor, and trace traffic flows for better insights.
Traffic Management: Optimize deployments with routing, shifting, timeouts, and circuit breaking.
Security: Ensure encrypted communication, granular access control, and fair usage policies.

By implementing these concepts, you can build resilient, scalable, and secure applications on Kubernetes with Consul. We hope this series has provided valuable insights to enhance your microservices architecture!