Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Level Up: Relic Solution Transition Guide for Distributed Tracing

distributedtracing

#1

How to Successfully Transition into Distributed Tracing

This guide will help set you up for success as you transition into using this new feature.

Features and Benefits

Modern applications and sites increasingly use many interconnected services. An application architecture that relies on many services or microservices is often referred to as a distributed system.
Distributed tracing is the process of tracking the activity resulting from a request to an application.

With distributed tracing, you can:

  • Trace the path of a request as it travels across a complex, distributed system
  • Discover the latency of the components along that path
  • Know which component in the path is creating a bottleneck :champagne:

The more distributed your system is, or the more distributed it is on the path to becoming, the more benefits you’ll get from New Relic’s distributed tracing feature. You’ll be able to filter traces by specific attributes—like duration or host name—and use Insights to query your data or make custom charts.

Learn more about benefits of this New Relic feature here in our documentation.

Use case: Solve latency issues quickly in a distributed system

Here’s an example of using New Relic’s distributed tracing feature to solve a latency problem:

  1. An engineer receives an alert notification showing that their service is experiencing frequent high latency.

  2. The engineer goes to the APM Overview page and sees a large increase in response time from dependent services.

  3. They go to the distributed tracing UI page and apply a filter for duration > 2, so they can look at only the traces for requests over that duration.

  4. They examine one trace with unusually long latency. They see that there is a call being made to a dependent service that is querying a database that is responding very slowly. They look at other slow traces during the time period and find that all the slow requests have the slow database as the common source of latency.

  5. They check with the owner of that service and learn that their team is currently working on scaling the database due to unexpected high throughput.

Compatibility and requirements

New Relic’s distributed tracing is available for APM Pro. It is currently available for these APM agents: Java, Node.js, Python, and Ruby. More agent support is currently underway.

NOTE: Enabling distributed tracing for an application impacts how other APM features work, specifically external services and service maps. Before enabling this feature, you should read the full transition guide.

Roll out distributed tracing

To get distributed tracing working, you must first update APM agents for any service you want to see details for in a distributed trace, and then enable them using APM agent configuration.

If you have a large system with many APM-monitored entities, you might first come up with a strategy for how to roll out distributed tracing across your system. Some ideas to consider:

  • Consider the requests that are the most important for your business, or the most likely to require analysis and troubleshooting. Determine the most important applications and services associated with those requests (perhaps using service maps), and update and enable the APM agents for those services.
  1. Enable tracing for the services at roughly the same time so you can more easily gauge the completeness of the end-to-end trace.
  2. When you begin to see traces for that request in the UI, you will see spans in the trace for external calls to other services. You can now target those services for APM agent install/upgrade to add additional details to the distributed trace.
  3. If an application is fairly standalone and not often used in context with other services, you may decide not to enable distributed tracing for it.

The image above represents one strategy for enabling distributed tracing across a large system with many New Relic agents.

Remember: you should read our more thorough transition guide in the docs before enabling distributed tracing.

Learn about UI features

See what the distributed tracing UI page looks like and what it can do.

Have thoughts or other use cases to share as you’re transitioning over? Let us know! We are eager to hear about your experiences. Feel free to reply below.


Distribution Tracing issue
Explorers Hub Treasure Hunt
Our Distributed Tracing Feature is Live!
#2

Hi, in the span attributes, can a custom attribute added which is app/microservice specific. This will help to link between the trace which took more time and what was the transaction from the application impacted. Also I have configured the feature in our production app which is deployed on PCF open stack. Is there any video walkthrough or detailed documentation exists on this new feature?


#3

@Sujit_Tripathy one of the benefits of Distributed Tracing is that it allows you to filter data by attributes, either default attributes common to microservices or custom attributes you’ve added.
While we can’t manually add custom attributes to spans or traces yet, you should be able to filter to any preexisting custom attributes found within your app i.e. contained in a trace. We don’t have heavy documentation on this subject and we are working on expanding these documents as questions like yours come along, so thanks for using this awesome feature :slight_smile:

However, we have provided some more details on the filtering feature on the document below and it is a good place to start:

Hope this helps.


#4

Thanks. We are using Pivotal Cloud Foundry with New Relic agent of v4.3. My question is is Distributed Tracing take the advantage of Spring Cloud Sleuth which does the similar feature for tracing? We have currently using sleuth but wondering can we take advantage of this feature integrated with sleuth like Zipkin? Any thoughts?


#5

@Sujit_Tripathy As far as I am aware we do not have an integration with Spring Cloud Sleuth or Zipkin as they are also other tools for tracking distributed tracing. We have this great blog post from Erika Arnold that explains the mysteries of tracing notion in general:

If you have any further questions on how distributed tracing could be integrated with your environment, I would highly recommend to reach out to your Account Executive as they have other resources that they could provide for you.

If you also have follow-up questions on this, please post them here and we be happy to answer.:upside_down_face: