Relic Solution: Hostname Explosions

Hostname Explosions

Have you ever had trouble loading custom durations for your charts in the APM user interface? Maybe you experience longer than usual load times when selecting large time periods in the time picker? These among other things are symptoms of an issue that we in support affectionately call: Hostname Explosions.

A Hostname Explosion is when an agent installed on an application generates a large number of unique Real Agent IDs, that when queried, causes issues in the APM UI. Real Agent IDs are used for tying an instance of a New Relic Agent to an application on a given host to the New Relic Collector. Unique Real Agent IDs are a symptom of spinning up an agent on a host with a dynamic name.

This normally would be a non-issue but when implemented on a large scale hundreds of thousands of these unique Real Agent IDs can get generated.

What can I do about this?

Adjust your hostnames to have uniform names. This doesn’t mean that every host used for your application needs to have the same name, it just means that there can’t be a unique hostname generated every time your application spins up. The reason for this is that the current system doesn’t scale well with hosts or containers with unique names, especially when frequently cycling containers and generating large amounts of short term IDs. The hostnames need to be set up to have names/IDs that don’t consistently change.

For example, if you’re using Docker, one option would be for you to use the --hostname flag with docker run:

Another example would be the hostname option for Kubernetes pods:

If you have an environment that is generating ephemeral hostnames and IDs and know of a way to adjust the hostname/ID, it would be very helpful for us and others if you could share them in the comments below.

Ok, I’ve done that but nothing changed?

While we understand it is not ideal, the Real Agent IDs generated still exist in the historical data of the application. As the previous data ages out the issues seen in the UI should improve. Here are the options you have regarding this:

  • Wait for the data to age out. This is 90 days from when the hostnames were changed.
  • You can change the reporting application name in your New Relic agent configuration. The historical data will remain on the previous app name for your review but you will need to look at the new reporting app name to see current and future data.
  • You can delete the current application by stopping it, deleting it in the UI, and then adding it again with the same name in the agent configuration. This is effectively the same as the previous option but without retaining historical data.

Important Caveat:

Most of the agents have a display_name option in their config, these do not change the reporting hostname. These simply change the displayed hostname in the UI.