How to Successfully Transition into Distributed Tracing
This guide will help set you up for success as you transition into using this new feature.
Features and Benefits
Modern applications and sites increasingly use many interconnected services. An application architecture that relies on many services or microservices is often referred to as a distributed system.
Distributed tracing is the process of tracking the activity resulting from a request to an application.
With distributed tracing, you can:
- Trace the path of a request as it travels across a complex, distributed system
- Discover the latency of the components along that path
- Know which component in the path is creating a bottleneck
The more distributed your system is, or the more distributed it is on the path to becoming, the more benefits you’ll get from New Relic’s distributed tracing feature. You’ll be able to filter traces by specific attributes—like duration or host name—and use Insights to query your data or make custom charts.
Use case: Solve latency issues quickly in a distributed system
Here’s an example of using New Relic’s distributed tracing feature to solve a latency problem:
An engineer receives an alert notification showing that their service is experiencing frequent high latency.
The engineer goes to the APM Overview page and sees a large increase in response time from dependent services.
They go to the distributed tracing UI page and apply a filter for
duration > 2, so they can look at only the traces for requests over that duration.
They examine one trace with unusually long latency. They see that there is a call being made to a dependent service that is querying a database that is responding very slowly. They look at other slow traces during the time period and find that all the slow requests have the slow database as the common source of latency.
They check with the owner of that service and learn that their team is currently working on scaling the database due to unexpected high throughput.
Compatibility and requirements
New Relic’s distributed tracing is available for APM Pro. It is currently available for these APM agents: Java, Node.js, Python, and Ruby. More agent support is currently underway.
NOTE: Enabling distributed tracing for an application impacts how other APM features work, specifically external services and service maps. Before enabling this feature, you should read the full transition guide.
Roll out distributed tracing
To get distributed tracing working, you must first update APM agents for any service you want to see details for in a distributed trace, and then enable them using APM agent configuration.
If you have a large system with many APM-monitored entities, you might first come up with a strategy for how to roll out distributed tracing across your system. Some ideas to consider:
- Consider the requests that are the most important for your business, or the most likely to require analysis and troubleshooting. Determine the most important applications and services associated with those requests (perhaps using service maps), and update and enable the APM agents for those services.
- Enable tracing for the services at roughly the same time so you can more easily gauge the completeness of the end-to-end trace.
- When you begin to see traces for that request in the UI, you will see spans in the trace for external calls to other services. You can now target those services for APM agent install/upgrade to add additional details to the distributed trace.
- If an application is fairly standalone and not often used in context with other services, you may decide not to enable distributed tracing for it.
The image above represents one strategy for enabling distributed tracing across a large system with many New Relic agents.
Remember: you should read our more thorough transition guide in the docs before enabling distributed tracing.
Learn about UI features
See what the distributed tracing UI page looks like and what it can do.
Have thoughts or other use cases to share as you’re transitioning over? Let us know! We are eager to hear about your experiences. Feel free to reply below.