I would like to make a chart on a panel to record the percentile of response time for an external service. This is what I write for showing averages, but how can I get it to show a chart showing multiple lines of different percentiles?
SELECT average(`apm.service.external.host.duration`) FROM Metric WHERE appName LIKE '%' AND `external.host` = '[SOME_IP]' SINCE 30 MINUTES AGO TIMESERIES
See screenshot of where this is used (it’s on a widget on the dashboard)
ok — but is there anyway I can create the equivalent of a percentile graph?
Let me put it this way, if I can do this in Kibana very easily then I think that an APM platform like New Relic ought to be able to do this.
You are able to visual this exact external call data in Kibana? Or are you talking in general that Kibana can visualize percentiles?
NR can visualize percentiles for metrics sources from event data. Data being surfaced using the Metric type isn’t actually event data. That can be a little confusing for new users and its also a little frustrating for everyone.
This thing you are trying to do is something we all want. Treat external calls as events. There are ways to get there with the solution.
You can try using externalduration for your transactions if you fully understand your external calls and you don’t have more than one from a transaction and no retries etc.
Well, I send the response time from my app to ES and then display this in Kibana. Since NRQL also collects that data, I figure that there should be a way to display it. So technically, yes, it is that exact external call data.
Obviously the implementation is different. My goal with NR is to replace the need for me to write custom things for everything that I need. I want to know how I can do that in NR. That’s why I asked how I can do something similar.
But you are right, I don’t know how the metrics is different than event — even if they are sampled, why can I track:
BUT NOT percentile?
Believe me. I understand.
There are some technical reasons why a metric aggregated at the host has difficulty with percentile and maybe why NR has chosen at this time not to include it. Just totally guessing. They could also be short staffed or have other priorities.
With an event you have all the data on the NR side and each event is more often a representation of a single transaction with it’s timing. You then query that data and aggregate in realtime using NRQL etc. You have total control with event data and its accurate*
Metric data is aggregated at the agent or host most of the time and you do not have per transaction data or attributes to query. Because the monitors are distributed you have some challenges. I’m maybe less understanding of the lack of percentile data for any metric derived from events that are still present in NR.
For percentiles that might be difficult to aggregate across thousands of hosts. You need to approximate, use math, some voodoo or witchcraft.
I was going to go into more detail, but I decided I could maybe find an article that explains it better than I can. The code is there. Maybe NR can give it a whirl.
