JMX - Host Integration - No Metrics

Hi NR Forum,

I’m working on getting the on-host integration JMX (nri-jmx) added onto a few of our Java servers. I’ve followed the documentation on installation and setup. (https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/jmx-monitoring-integration)

Steps:

  1. Infrastructure agent is installed on server and running (infra metrics are showing in infrastructure)
  2. Install nri-jmx agent on server
  3. cp jmx-config.yml in /etc/newrelic-infra/integrations.d
  4. Update config file
integration_name: com.newrelic.jmx

instances:
  - name: jmx
    command: all_data
    arguments:
      jmx_host: jmx-host.localnet
      jmx_port: 9999
      jmx_user: admin
      jmx_pass: admin
      collection_files: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml"
    labels:
      env: staging
      host: staging-myserver-01a
      group: staging-myserver
  1. Restart service + restart newrelic-infra agent
  2. Monitor infrastructure on-host integration JMX - nothing is coming up

Is there something I am missing that is stopping the application from coming up in infrastrcutre - jmx? Do I need a service account for jmx_user and jmx_pass?


Edit 1: I’ve made some changes but still no luck on getting jmx monitor up in infrastructure.

I’ve created a service account specific for staging application

jmx_user: myuser
jmx_pass: { my_password}

I’ve update the JVM options for application to allow monitoring on jMX port 9999

-server -Xmx12G -Xms12G -Xmn3G -XX:+UseLargePages -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Hey @ian.arsenault - Taking a look at the JVM arguments you provided, I don’t think you’ll need to have a username/password configured here. The JMX port can have it’s own login information separate from the service itself, but this is typically defined with JVM arguments that I don’t see here. Instead it looks like you’re currently setup for JMX with no authentication (see this link).

Assuming the host and port are correct, could you try removing the jmx_user and jmx_pass configurations to see if that helps?

One other useful command that can be used for quickly testing various configurations is:

echo "*:*" | /usr/bin/nrjmx -hostname localhost -port 9999 -username admin -password admin --verbose true

In this case you would want to try running that command without the username and password arguments first; I just wanted to show how that could be used in case it helps with future testing. What this command does is pass wildcards for both domain and bean name to the integration so that it outputs all available data…if it can connect. If it can’t connect you should see an error stating as much. You can also pipe specific domain:bean names to the integration to see if specific data is available such as:

echo "java.lang:type=Threading" | /usr/bin/nrjmx -hostname localhost -port 9999 -username admin -password admin --verbose true

Currently, if that data isn’t available, you will just get an empty response such as [](future Explorer’s please note this should be changing shortly to give more verbose/useful output!)

These commands are just good for quickly testing configuration options against the integration and not needing to hunt for errors in the infrastructure agent logs. I hope that helps, let us know how it goes!

1 Like

@MrMatt Thanks for the response. This is definitely helpful in debugging the issue.

I’ve tried running the command with and without username + password

echo "*:*" | /usr/bin/nrjmx -hostname localhost -port 9999 --verbose true 

and receive the following error

Oct 31, 2019 12:20:00 AM org.newrelic.nrjmx.Application main                                                                                                                                                                                
SEVERE: Can't connect to JMX server: service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi                                                                                                                                                    
Oct 31, 2019 12:20:00 AM org.newrelic.nrjmx.Application main                                                                                                                                                                                
FINE: Can't connect to JMX server: service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi                                                                                                                                                      
org.newrelic.nrjmx.JMXFetcher$ConnectionError: Can't connect to JMX server: service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi                                                                                                             
        at org.newrelic.nrjmx.JMXFetcher.run(JMXFetcher.java:161)                                                                                                                                                                           
        at org.newrelic.nrjmx.Application.main(Application.java:42)                                                                                                                                                                         
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:                       
        java.net.ConnectException: Connection refused (Connection refused)]                                                                                                                                                                 
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)                                                                                                                                                          
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)                                                                                                                                                
        at org.newrelic.nrjmx.JMXFetcher.run(JMXFetcher.java:158)                                                                                                                                                                           
        ... 1 more                                                                                                                                                                                                                          
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:                                                                               
        java.net.ConnectException: Connection refused (Connection refused)]                                                                                                                                                                 
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)                                                                                                                                                       
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)                                                                                                                                                    
        at javax.naming.InitialContext.lookup(InitialContext.java:417)                                                                                                                                                                      
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)                                                                                                                                               
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)                                                                                                                                                   
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)                                                                                                                                                          
        ... 3 more                                                                                                                                                                                                                          
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:                                                                                                                                           
        java.net.ConnectException: Connection refused (Connection refused)                                                                                                                                                                  
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)                                                                                                                                                                
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)                                                                                                                                                           
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)                                                                                                                                                              
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)                                                                                                                                                                           
        at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)                                                                                                                                                            
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)                                                                                                                                                       
        ... 8 more                                                                                                                                                                                                                          
Caused by: java.net.ConnectException: Connection refused (Connection refused)                                                                                                                                                               
        at java.net.PlainSocketImpl.socketConnect(Native Method)                                                                                                                                                                            
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)                                                                                                                                                     
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)                                                                                                                                              
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)                                                                                                                                                       
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)                                                                                                                                                                       
        at java.net.Socket.connect(Socket.java:589)                                                                                                                                                                                         
        at java.net.Socket.connect(Socket.java:538)                                                                                                                                                                                         
        at java.net.Socket.<init>(Socket.java:434)                                                                                                                                                                                          
        at java.net.Socket.<init>(Socket.java:211)                                                                                                                                                                                          
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)                                                                                                                                      
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)                                                                                                                                     
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)                                                                                                                                                                
        ... 13 more           

It’s throwing a “can’t connect to jmx server” error for both commands, which indicates i’m missing another piece, possibly another JVM configuration? I’m not too sure as this area is new to me but I will keep investigating. If you or your team has any further insight, it would be greatly appreciated.

Thanks
Ian

Hi @ian.arsenault! I am just checking in after a week to see if you made any headway with this no-metrics issue? We would love to hear an update on how things are going. Please let us know if there is anything we can do to further help. :blush: Talk soon!

I’m running into the same issue. Would be interested to know if/how this was resolved.

This is still being worked on at the moment. Unfortunately other things have taken priority but I will resume again hopefully soon. I will post my solution as soon as discover. If you figure this out as well, please post!

Ian

Looking forward to see what you come up with Ian :smiley:

I am getting the same error.
I cant see the dashboard in my newrelic infra

Hello @ankitkumarsrivastava,

Please change your Infra agent logging level to verbose and let us know the output:

https://docs.newrelic.com/docs/infrastructure/infrastructure-troubleshooting/troubleshoot-logs/generate-logs-troubleshooting-infrastructure

There should be a call to the JMX integration there that may shed some light on the issue.