Tools and Patterns
There are a number of modern tools to make systems observable. While identifying and/or creating tools that work for your system, here are a few things to consider to help guide the choices.
- Must be simple to integrate and easy to use.
- It must be possible to aggregate and visualize data.
- Tools must provide real-time data.
- Must be able to guide users to the problem area with suitable, adequate end-to-end context.
Choices
Service Mesh
Leveraging a Service Mesh that follows the Sidecar Pattern quickly sets up a go-to set of metrics, and traces (although traces need to be propagated from incoming requests to outgoing requests manually).
A sidecar works by intercepting all incoming and outgoing traffic to your image. It then adds trace headers to each request and emits a standard set of logs and metrics. These metrics are extremely powerful for observability, allowing every service, whether client-side or server-side, to leverage a unified set of metrics, including:
- Latency
- Bytes
- Request Rate
- Error Rate
In a microservice architecture, pinpointing the root cause of a spike in 500's can be non-trivial, but with the added observability from a sidecar you can quickly determine which service in your service mesh resulted in the spike in errors.
Service Mesh's have a large surface area for configuration, and can seem like a daunting undertaking to deploy. However, most services (including Linkerd) offer a sane set of defaults, and can be deployed via the happy path to quickly land these observability wins.