Why does the apm.service.datastore.operation.duration need "x 1000"?

Hi all;

I try to use apm.service.datastore.operation.duration to monitor the operations on a database table. I started to look at the chats in APM > Databases like " Top database operations", in the query, it has something like:

SELECT sum(apm.service.datastore.operation.duration) * 1000 …

Why “* 1000”?

The y-axis label in this chat shows “s” (second) but If I run the query in the query builder, the y-axis becomes 1000 times larger. E.g., 3s in the original chart becomes 3k in query builder! Is this “x 1000” only for the display in the pre-defined chart?

The bottom line is: what is the “unit” in this metric? Second? Millisecond?

Cheers,
KC

Hi @Keith.Cheng Sorry its taking awhile to get a response on your question. I’m going to reach out to @philweber and @RyanVeitch to see if they can clarify what the “unit” means in your example. Thanks for your patience!

@JoiConverse, thanks for following it up.

Hi @Keith.Cheng, I’d be inclined to believe the metric for apm.service.datastore.operation.duration is seconds. The reason being is that that the use of *1000 is often used by people who simply want this value to be represented in milliseconds so they’re manually converting the metric to suit whatever their needs may be. However, I would still like to defer to @philweber and @RyanVeitch just in case I’m wrong.

I have a feeling that this is a bug in the ‘View Query’ for that chart. I took a look at it & by running the query:

SELECT sum(apm.service.datastore.operation.duration) FROM Metric WHERE (entity.guid = 'myEntityGUID') FACET `metricTimesliceName` LIMIT 5 SINCE 24 HOURS AGO TIMESERIES

I get a chart that matches the one in the APM Databases page. So - the * 1000 part of it feels unnecessary.

It may be that this is a standard where most metric charts are in milliseconds, so the *1000 brings it to seconds. But in this case the data is already in seconds, so multiplying by 1000 is unnecessary.

I’m going to report this internally and find out more info - I’ll follow up here if I find anything that is different to the above info.

@RyanVeitch: If the metric is in milliseconds, wouldn’t / 1000 bring it to seconds?

Yes but the query is multiplying not dividing

I have heard back from the team internally - they confirm that the metric is recorded in ms.

So the APM chart shows Xs or Yms for seconds or milliseconds.

When you run the associated query, the number will look different but that is since it the query results can not determine what the metric represents (in this case time) - so it can’t round into a nicely viewed seconds or millisecond value.

So in the APM UI where you see that metric at 12s, running the query will show 12000, this is milliseconds.

Thanks everyone. I understand that the chat in the APM can render to “second” but in query builder, it just shows the figures without any “unit” assumption.

However, I am still confusing: if the metric is in ms, why do I want to x 1000? As @philweber pointed out, if we want to show the result as “second”, it shall be “/ 1000”, not times.

More, if I run the query without “x 1000”, in theory, it will give me “ms”, hence in this case, I have a query peak at ~2000 ms, ie. around 2 sec in the following “query builder” chat:

query builder.png1551×634 19.7 KB

However, in the APM chart, it shows 2000 seconds

APM db query chart.png1080×238 8.46 KB

Not only the chat but the “slow queries” table also shows 36 min (i.e., 2160 sec).

So, which one is correct? The query builder (without x 1000) 2000 ms or the APM, 2000 seconds?

Thanks.