Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

@Trace annotation and 99th percentile

api

#1

Hello,
I’m relative new to New Relic, I’m trying to get the 99th percentile of a transaction, to be precise of something that happens during a transaction.

There’s an endpoint of the application backend (we use Tomcat) that it is called by the application. This endpoint will be monitored by default and it will be possible to get the 99th percentile for this endpoint. My problem is that a lot of different things are happening when this endpoint is called but I’m only interested in knowing the 99th percentile of a particular method called during the endpoint set of executed instructions (for this method I can see some information in the breakdown table, like avg calls ad avg time). I suppose that it’s not possible to get this information using insight and the query language.

So I tried to find another solution by using the @Trace annotation:

@Trace(dispatcher=true)
@Override
protected void run() {
    // stuff
    externalRestfulService.callSomeMethod();
    //other stuff
}

I want to point out that the external service is on New Relic as well but has only one transaction, it was created so that all the calls are redirected to the same endpoint, long story short it is not useful for the 99th percentile.

The run method is an hystrix command overridden method. I want to find out the 99th percentile of this run method as a whole (stuff and other stuff included). The problem is that even after adding the annotation, I can’t seem to find the run method in the transaction list of New Relic (enable_custom_tracing has been set to true in the newrelic.yml file).

Do you have any idea or suggestion?


#2

@luca.rocket - So if your application is in the middle of a transaction when a method decorated with @Trace is encountered, the New Relic agent will not create a new transaction so I suspect that is why you are not seeing the data you expect. One way of working around this particular issue would be to add a manual timer around your method you’re interested in and then adding that data to the existing transaction as a custom attribute. Once that data is added to the existing transactions you can use Insights to return the 99th percentile directly (Insights percentile function)

Does that match what you are trying to accomplish or did I misunderstand what you are trying to accomplish?


#3

Hi! This is an interesting workaround (and yes, this is what I wanted to accomplish). It’s a shame I can get a lot of information (like avg/min/max response time, standard deviation, …) for that particular method in Insight->Data Explorer->Metrics, but not the percentile. Also, I thought that since some information about that method is stored, it would be possible to run queries on Insights. Anyway I hope I’m not duplicating data by timing the method execution (since I suppose New Relic stores something similar already for computing avg/min/max, …). The main question about this solution is: how will this affect performance? I suppose the impact is very low, it should only add the key/value pair to the values that will be reported for this transaction, nothing complex or call to services, am I correct?


#4

@luca.rocket - So for each transaction we record overall metrics for the transaction as well as metrics for each transaction segment. Those metrics are recorded by the New Relic APM agent in aggregate each minute which is why it’s unfortunately not possible to reconstruct the 99th percentile after the fact using the metrics. The difference with the Insights events is that we record every (generally) transaction as an event so it becomes a lot more flexible when you are wanting to do things like calculate the 99th percentile after the fact.

As far as the impact to adding this custom attribute should be pretty minimal. This behavior of timing each method is basically how the APM agents themselves work. Unfortunately since an individual transaction may be composed of several hundred different methods, we don’t add individual timing of every method to every Insights transaction event which is why to meet your particular use case, this seems like the easiest way to solve your particular situation.


#5

Continuing the discussion from @Trace annotation and 99th percentile:

I added @Trace(metricName=“Test”) to one of the method which process the kafka messages. I tried to pull the metrics from SpringBootSample but I do not see the metricName = ‘Test’. “enable_custom_tracing” is enable . Can some one help me why I am not seeing the custom metricName in Newrelic.


#6

Can you send a link to the app in which your test metric should exist?