[Node] Pg information/queries not displaying in APM

[jstromwick_xp]

Agents question template

Hello! I recently setup my node app with NewRelic. Metrics are coming through fine, except I can’t see any database query information. I’m using pg to connect to postgres db. I’ve run the agent locally (windows 10) and in our production environment (linux) Would love some troubleshooting!

versions:
newrelic agent: 8.7.1
node: 8.3.2
express: 4.17.2
pg: 8.7.1

Configuration is using the following environment variables:
NEW_RELIC_ENABLED=true
NEW_RELIC_LOG_LEVEL=“debug”
NEW_RELIC_APP_NAME="—"
NEW_RELIC_LICENSE_KEY="—"
NEW_RELIC_NO_CONFIG_FILE=true
NEW_RELIC_LOG_ENABLED=true
NEW_RELIC_AUDIT_LOG_ENABLED=true
NEW_RELIC_RECORD_SQL=“obfuscated”
NEW_RELIC_SLOW_SQL_ENABLED=true
NEW_RELIC_ALLOW_ALL_HEADERS=true
NEW_RELIC_ATTRIBUTES_EXCLUDE= ‘request.headers.cookie,request.headers.authorization,request.headers.proxyAuthorization,request.headers.setCookie*,request.headers.x*,response.headers.cookie,response.headers.authorization,response.headers.proxyAuthorization,response.headers.setCookie*,response.headers.x*’

Link to my New Relic instance:
https://one.newrelic.com/nr1-core/apm-nerdlets/databases/MzM5MjI4NHxBUE18QVBQTElDQVRJT058MTA0NTA1MDY1NQ?account=3392284&duration=1800000&state=809e19a4-968f-6306-902e-596b42eb4ee1

[eschwall]

Hey there @jstromwick_xp ,

Thanks for reaching out to the New Relic Community! I think I might be able to shed some light on your situation through this doc HERE

If you have any questions or extra details to add in the meantime, then please don’t hesitate to reach out.

Cheers

[jstromwick_xp]

Thanks eschwall! Read through those docs, makes sense and I can take a crack at using custom instrumentation to get at what I want.

That being said, I was under the impression that pg was automatically instrumented by the new-relic node agent. Was that just a faulty assumption on my part, or is it something else, like the fact that we’re using async/await?

[sbedi]

I’d like to chime in here. I believe pg is auto instrumented, but it looks like your versions of node and express are old and unsupported per our compatibility standards.

I would recommend both of these to the most recent stable version (please note that although Node 12 is currently supported, it will be deprecated in April).

[jstromwick_xp]

Thanks sbedi! Good call out.

I recently updated our versions for node/express, new app versions are:

node: 16.13.2
express: 4.17.2

That being said, I am still not seeing any data in the databases section…

Would totally appreciate any help pointing me in the right direction as to what to look too next!

[sbedi]

@jstromwick_xp -

Can you check your version of Express? We support Express 4.6.0 or higher.

[jstromwick_xp]

Thanks for the reply!

In production express is @ 4.17.2

[sbedi]

@jstromwick_xp -
This is an unsupported Express version; we are continually making agent improvements, and as such, older Express versions may exhibit unexpected behavior, such as not reporting database transactions. Can you upgrade to the at least Express 4.6.0?

[jstromwick_xp]

Totally get the need to be on the latest Express (and any other) package versions.

Our app is running on Express 4.17.2

Express 4.17.2 was released 12/21/2021
Express 4.6.0 was released 7/11/2014

express

My understanding from reading the compatibility guidelines you linked above is that Express versions greater than 4.6.0 should be supported by the node agent. Express 4.17.2 is a greater version than Express 4.6.0

Appreciate the help so far and apologies if I’m missing something.

[sbedi]

Thanks for clearing that up about Express!

Are you using Babel with your application? It could potentially rearrange the order that the modules load, thus causes some modules to not be instrumented. New Relic has to be the first module in order to instrument your code properly, one method to reinforce this is using the -r require flag to preload the newrelic module at startup using the Node command line option -r.

For example, if your application’s entry point was ./dist/server.js then your start up script would be:

node -r newrelic ./dist/server.js

Please Note: Even when preloading the newrelic module, you will still need to call the module at the top of your main file with require('newrelic'); or import newrelic from 'newrelic'

[jstromwick_xp]

Thanks for the instruction, and sorry for the long delay! I’m not using babel, but am using esbuild (https://esbuild.github.io/) to transpile from Typescript to Javascript.

I had been using node -r with new relic, but had not actually added the import to server.ts. However after adding that I’m still not seeing any DB transactions.

Is there a log file or anything I could inspect or share to give some more info?

[dcody]

Hi @jstromwick_xp

Thanks for reaching back out with your findings. I will need to loop in the engineer team here as this is out of my scope.

Please note they will post their finding here, should you have updates or questions please feel free to reach out here.

[bhensley]

Hi jstromwick_xp,

I’m Brian from Node support, thanks for all the info here! In your first post you list the env var NEW_RELIC_RECORD_SQL="obfuscated" but I noticed in your application’s environment page that value is set to off:

I think something is going wrong with the setting of that variable which may be the cause of your issue. Could you try configuring it in some other way, like a newrelic.js to see if the value will update to obfuscated?

Thanks

-Brian

[jstromwick_xp]

Thanks for the follow up Brian!

It looks like I hadn’t added NEW_RELIC_RECORD_SQL locally or on production. I added that env variable NEW_RELIC_RECORD_SQL=“obfuscated” locally and on production, and I still didn’t see any DB traces or info.

I see the updated variable on the environment page for my local instance, but don’t see it update on the production instance. I double checked the variable name and value to make sure I hadn’t missed anything. Would it take a while for that to update?

Edit: Looks like the field is now showing up as obfuscated, but I still don’t see any DB traces

Thanks!

–Jesse

[dcody]

Hi @jstromwick_xp

So sorry to hear you still facing this issue.

While this is out of my scope, I will be sure to get you the support needed here. I have gone ahead and looped in out Node Agents Engineer here.

They will reach out here with their findings. Should you have any updates or questions, please reach out.

[bhensley]

Hi jstromwick_xp,

Can you try to get me some trace level logs while your app is running and making database calls. Here are instructions on how to do that. Feel free to send the newrelic_agent.log to me in a direct message.

Could you also tell me which of the transactions your app is making contain database calls? It doesn’t have to be all, but if you can point to one or two you know if accessing your database that will help me.

Thanks,

Brian

[bhensley]

Hello again jstromwick_xp,

Thank you for sending me the log file. Looking it over it doesn’t appear to be instrumenting any database transactions at all, I’m going to look into if this could be related to using esbuild and I’ll get back to you. I spoke with the engineers about your issue and we have a couple more questions.

• What HTTP library are you using that makes requests to this express app?
• Is it possible to make a simplified reproduction of your app for us to test with? Alternatively if this is posted in a public github repo and you could provide a link that would work. Feel free to DM that instead of posting it publicly.

Thanks
-Brian

[jstromwick_xp]

Interesting…

For requests to the api, one front end is using fetch, the other is using axios.

I can work on trying to get a simplified version of our api in a public repo…

thanks for the help!

-Jesse

[bhensley]

Thanks for all the info you’ve provided jstromwick. I looked into esbuild some and it looks to me like this is very likely your problem. The node agent doesn’t play well with JavaScript bundlers, but we have bit of a workaround using instrumentLoadedModule in the Node API. Have you seen this post about the problems with bundlers and how to use the workaround yet?

• The Node.js APM Agent and Webpack

Best,

Brian

[jstromwick_xp]

That looks to be what was going on! After adding the code below and calling it before starting express I can now see both express and postgres data in new relic!

Thank you for the help on this! I really appreciate it!

Adding the code below in case it helps anyone else

import newrelic from "newrelic";
import express from "express";
import pg from "pg";

export function initNewRelic() {
  newrelic.instrumentLoadedModule("express", express);
  newrelic.instrumentLoadedModule("pg", pg);
}

then call this before starting express in my server.ts

import { initNewRelic } from "./utils/newRelicInit";    
import express from "express";    

initNewRelic();
const port = 8888;
const app = express();

//...rest of file

I also have newrelic setup as an external package on build and require it on run. Scripts in my package.json looks like:

 "scripts": {
    "server:build": "esbuild server/server.ts --target=node16 --outfile=bin/server.js --platform=node --bundle --external:pg-native --external:newrelic --sourcemap",
    "server:start": "node --require newrelic bin/server.js",
 }