[Java] New Relic causes WildFly to stop immediately due to LogManager issue

We try to run New Relic on WildFly 19 + Java 11 + Windows 2016 server. Adding the Java agent causes WildFly to terminate immediately. Troubleshooting was made difficult by the fact that WildFly did not produce any error logs. After some digging, the root cause seemed to be that installing New Relic agent messes up logging in WildFly and that causes WildFly to die. Running the instance from terminal reveals error Could not load Logmanager “org.jboss.logmanager.LogManager”. Some googling returns a number of relevant results indicating various tricks that can be tried (for instance adding -Djava.util.logging.manager=org.jboss.logmanager.LogManager or tweaking bootclasspath) but none of these resolved the issue.

It seems that this is somewhat generic. It also seems that the detailed installation instructions New Relic already provides are not sufficient to run New Relic on WildFly.

Hello Tuukka!

We’ve seen this issue before, in fact there’s a discussion of it here as well. As you said It has to do with the Jboss logmanager. It appears that JMX needs the logging classes before the proper modules are loaded by the JBoss/Wildfly classloader. Thus they need to be made available on the bootstrap classloader for earlier access.

Basically the workaround is to update the wildfly-n.n.n/bin/standalone.conf file as follows.

A) The first thing is to add certain packages to the JBoss system module loader described here: Jboss Modules - Loader

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager,com.newrelic"
fi

B) Then at the bottom of the standalone.conf add the following config that sets the logmanager and wildfly-common jars on the bootstrap classpath as well as setting the java.util.logging.manager:

This is assuming that the jmx and Java agent config already exist and it’s really just the config that sets the logmanager and wildfly-common * jars on the bootstrap classpath as well as the java.util.logging.manager that would need to be added.

# wildfly install directory (Add your actual path here)
JBOSS_HOME="/path/to/wildfly-19.1.0.Final"

# enable jmx remote
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.rmi.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

# logmanager jar (update jar version as needed)
JBOSS_LOG_MANAGER_LIB="$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.14.Final.jar"

# wildfly-common jar (update jar version as needed)
WILDFLY_COMMON_LIB="$JBOSS_HOME/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.5.1.Final.jar"

# register jboss logmanager at jvm early startup and add the logmanager and wildfly-common libs to the bootstrap classpath
# For Java 8 and below use -Xbootclasspath/p, for Java 9+ use -Xbootclasspath/a
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/a:$JBOSS_LOG_MANAGER_LIB:$WILDFLY_COMMON_LIB"

# Might be needed, see: https://issues.jboss.org/browse/WFCORE-1494
# -Dsun.util.logging.disableCallerCheck=true

# Might be needed
# -Dorg.wildfly.logging.skipLogManagerCheck=true

Note: For Java 8 and below use -Xbootclasspath/p (the prepend option) and for Java 9+ use -Xbootclasspath/a (the append option) - more about that here: Oracle Java 9 release notes

Note the two lines that are commented out. The disableCallerCheck might be needed.

Let me know if that works! :slight_smile:

2 Likes

Hi Giselle,
thank you very much for your help. I think I had tried virtually all of the additional config lines in different order and combinations, but you were able to find the correct to apply those. As note to myself and others perhaps finding this useful, the config line to set the javaagent in the original WildFly doc must be included as well.

We have now data flowing in and we can start exploring.

Best regards,
Tuukka

2 Likes

Hi Tuuka,

That’s great that the work around solved the issue! I also found out that the latest Java agent version 6.1.0 has a fix for this issue. in case you’d want to upgrade!

Cheers,
Giselle

Hi, I managed to get Java agent working with version 6.0 so I am happy not to fix something that is not broken.

I noted that WildFly 21 is out and the configuration file (for WIndows) has the following lines added at the bottom:

rem # Uncomment to add a Java agent. If an agent is added to the module options, then jboss-modules.jar is added as an agent
rem # on the JVM. This allows things like the log manager or security manager to be configured before the agent is invoked.
rem set “MODULE_OPTS=-javaagent:agent.jar”

This might cause some changes in how New Relic agent should be configured.

Best regards,
Tuukka