403 Forbidden delta source not allowed Error

New Relic Infrastructure Question Template

  • Please provide a Permalink so we can see the exact time and place you are:

  • Which version of Windows or which distribution of Linux are you using?
    Ubuntu 18.04.3

  • What version of the Infrastructure agent are you using?
    1.8.2

  • What is your Infrastructure Subscription level? Essentials or PRO?
    Essentials

  • Describe what you are seeing. How does this differ from what you expected to see?
    When trying to run the newrelic-infra agent, we see this error when it tries to post its state:
    time="2019-12-09T20:23:15Z" level=error msg="couldn't post state" component=stateSender entityKey=unique-hostname-1 error="InventoryIngest: state change not accepted: 403 403 Forbidden {\"description\":\"delta source not allowed\",\"error\":\"error\"}\n" postStateResults="<nil>"
    This agent is freshly installed with only the license key and logging specified in the config. We’re not sure how to interpret this error message and couldn’t find it referenced anywhere on this help portal.
    We have the agent already running on several machines successfully, but they are using much older agent versions and have been running for a long time.

Hello @Sean.OBrien1, That seems strange. Could you confirm if you are using the correct license key in the config file?

I can confirm that the license key on the machine where this is failing is the same as the license keys on the machines where this is working. Using an invalid license key results in a different error (401 Unauthorized).

Hi @Sean.OBrien1

You’re correct about malformed license_keys resulting in 401 errors rather than 403. That error message seems to indicate that an inventory delta is being rejected by the Infrastructure API, implying that the agent was able to initialize and validate its license_key. However, the key-value appears to be error:error, which is not part of the default Infrastructure inventory namespace. Do you have this agent configured with an on-host integration? Integrations will only work with an Infrastructure Pro license, and your Essentials license may be resulting in that message.

1 Like

Hi @sellefson. We have no integrations at all. This is a completely clean install (apt purged, deleted leftover etc config, reinstalled with 1.8.2, repopulated license key). If there is any other configuration that needs to be manually deleted I would be happy to do that.

The /etc/newrelic-infra.yml is:

license_key: <redacted>

log_file: /var/log/newrelic-infra/newrelic-infra.log

log_to_stdout: false

The newrelic-infra directory in etc contains an empty integration.d directory:

some-user@some-hostname:/etc# tree -a newrelic-infra
newrelic-infra
└── integrations.d

Hi @Sean.OBrien1,

My next suggestion would have been to remove the /var/db/newrelic-infra/data directory to try and clear any potentially malformed payloads that were cached by the agent, but apt purge newrelic-infra and reinstalling would have already taken care of that.

To get to the bottom of this, I believe we’ll need to review some verbose logs from the Infrastructure agent. I am going to create a ticket to follow-up with you and collect that information.

Hi @Sean.OBrien1,

Thanks for working with me in that ticket. I wanted to add a note here to help others who may encounter the issue you did.

The host_aliases plugin which populates the inventory under metadata/host_aliases is showing the agent is picking up localhost for the fullHostname . Versions of the agent after v1.2.6 started resolving DNS to determine what the FQDN is. If a host is identified as localhost , this can cause multiple agents to report under one entityId (a unique identifier assigned by the New Relic backend to identify individual hosts). Accordingly, the Infrastructure agent requires a unique hostname for each agent. There are a couple of approaches that can be taken here:

  1. Set a display_name in your newrelic-infra.yml , which will override the value used to determine the entityId
  2. Disable the DNS resolver by setting dns_hostname_resolution: false in your newrelic-infra.yml , which will revert the agent’s behavior to looking at the output of hostname -f
1 Like