When designing modern microservice architectures, service-to-service communication is integral. Kubernetes provides basic load-balanced routing through Services, and basic network security through Network Policies (if the Container Network Interface supports them!), but oftentimes this just isn't enough, not to mention it can be challenging to monitor and troubleshoot communication. This is where service meshes come in.

A service mesh was one of the first pieces we wanted to incorporate in our Eagle Platform. The key features we were looking at were:

• Traffic control
• Secure communication
• Metrics, logs, and traces
• Policy enforcement
• Ability to extend the service mesh

When looking into service meshes the three names that come up over and over again are Istio, Linkerd, and Consul. There are many sites and blogs out there comparing these service meshes, but the best one I've found is servicemesh.es. All three of these service meshes are very feature rich and have their pros and cons.

‍

Choices, choices, choices...

We like how simple and light weight Linkerd is. Pound for pound it is definitely a strong contender. For us, the limiting factors were not being able to extend the service mesh beyond Kubernetes and the somewhat lesser routing options.

Consul Connect is more complex, but gives us the ability to extend the service mesh to include virtual machines (VMs) and containers outside Kubernetes. When we were first starting Eagle, Consul didn't have an ingress solution, but they have since added an api-gateway. Consul isn't quite as Kubernetes-native, having separate Application Programming Interfaces (APIs) instead of using custom resource definitions (CRDs). It also requires a client running on each node in the cluster.

Istio is the most popular service mesh, and with good reason. It is the most feature rich, including things like rate limiting, fault injection, and delay injection that the others don't. With this, of course, comes added complexity.

‍

Picking one

In the end, it came down to the features Istio provides and the ecosystem around it. We loved all the capabilities it provided but didn't want to lock into the complexity for all projects. We support projects that might have only a handful of services running and don't necessarily need a ServiceMesh.

For Eagle we wanted the capability to leverage all the features of Istio for projects that need them, but didn't want to add complexity and overhead to simple projects that only need the Kubernetes native Service handling. So, we designed the platform around the ability to selectively enable Istio with a single flag. That way we can have the best of both worlds.