Instrumenting Azure AKS: agent ingesting failing with 401

Hi, Is the 401 error needs to be open up from azure NSG? I’m using new relic account with license key and followed the instruction from “Set up Kubernetes” AKS page.

I used helm to deploy the new relic helm chart. All pods discovered successfully in AKS.

Logs from new relic infrastructure pods,

time="2021-01-21T01:05:09Z" level=info msg="runtime configuration" agentUser=root component="New Relic Infrastructure Agent" executablePath= maxProcs=1 pluginDir="[/etc/newrelic-infra/integrations.d /var/db/newrelic-infra/integrations.d]"
time="2021-01-21T01:05:09Z" level=info msg="Checking network connectivity..." component=AgentService service=newrelic-infra
time="2021-01-21T01:05:10Z" level=info msg=Initializing component=AgentService elapsedTime=750.866726ms service=newrelic-infra version=1.14.2
time="2021-01-21T01:05:11Z" level=warning msg="Commands initial fetch failed." component=AgentService error="unsuccessful response, status:401 []" service=newrelic-infra
time="2021-01-21T01:05:11Z" level=info msg="New Relic infrastructure agent is running." component=AgentService elapsedTime=1.552136655s service=newrelic-infra
time="2021-01-21T01:05:11Z" level=info msg="Starting up agent..." component=Agent
time="2021-01-21T01:05:11Z" level=warning msg="failed to connect to DBus. make sure systemd is present." component=NotificationHandler
time="2021-01-21T01:05:11Z" level=warning msg="failed to init shutdown monitor" component=NotificationHandler error="no systemd found"
time="2021-01-21T01:05:11Z" level=info msg="Agent plugin" plugin=metadata/attributes
time="2021-01-21T01:05:11Z" level=info msg="Agent plugin" plugin=metadata/system
time="2021-01-21T01:05:11Z" level=info msg="Agent plugin" plugin=metadata/host_aliases
time="2021-01-21T01:05:11Z" level=info msg="Agent plugin" plugin=metadata/agent_config
time="2021-01-21T01:05:11Z" level=info msg="Agent plugin" plugin=metadata/proxy_config
time="2021-01-21T01:05:11Z" level=info msg="Integration info" arguments="map[]" command=metrics commandLine="[./bin/nri-kubernetes --metrics]" env-vars="map[CLUSTER_NAME:gls-aks KUBERNETES_SERVICE_HOST:10.0.0.1 KUBERNETES_SERVICE_PORT:443 NRK8S_NODE_NAME:aks-akspool01-64199536-vmss000006 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin VERBOSE:0]" instance=nri-kubernetes integration=com.newrelic.kubernetes interval=15 labels="map[]" os=linux prefix=integration/com.newrelic.kubernetes protocolVersion=2 workingDir=/var/db/newrelic-infra/newrelic-integrations
time="2021-01-21T01:05:11Z" level=info msg="Integration health check starting" instance=nri-kubernetes integration=com.newrelic.kubernetes prefix=integration/com.newrelic.kubernetes working-dir=/var/db/newrelic-infra/newrelic-integrations
time="2021-01-21T01:05:11Z" level=info msg="Integration health check finished with success" instance=nri-kubernetes integration=com.newrelic.kubernetes prefix=integration/com.newrelic.kubernetes working-dir=/var/db/newrelic-infra/newrelic-integrations
time="2021-01-21T01:05:12Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:05:13Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:05:14Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:05:17Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:05:23Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:05:30Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:05:50Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:06:11Z" level=warning msg="commands poll failed" component=CommandChannelService error="unsuccessful response, status:401 []"
time="2021-01-21T01:06:43Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:07:11Z" level=warning msg="commands poll failed" component=CommandChannelService error="unsuccessful response, status:401 []"
time="2021-01-21T01:08:07Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:08:11Z" level=warning msg="commands poll failed" component=CommandChannelService error="unsuccessful response, status:401 []"
time="2021-01-21T01:09:11Z" level=warning msg="commands poll failed" component=CommandChannelService error="unsuccessful response, status:401 []"
time="2021-01-21T01:10:11Z" level=warning msg="commands poll failed" component=CommandChannelService error="unsuccessful response, status:401 []"
time="2021-01-21T01:10:19Z" level=warning msg="agent connect attempt failed" component=IdentityConnectService error="ingest rejected connect: 401 401 Unauthorized "
time="2021-01-21T01:11:12Z" level=warning msg="commands poll failed" component=CommandChannelService error="unsuccessful response, status:401 []"

Helm deployment:

helm repo add newrelic https://helm-charts.newrelic.com

helm install newrelic-bundle newrelic/nri-bundle \
 --set global.licenseKey=my_lic_key \
 --set global.cluster=gls-aks \
 --namespace=default \
 --set newrelic-infrastructure.privileged=true \
 --set ksm.enabled=true \
 --set prometheus.enabled=true \
 --set kubeEvents.enabled=true \
 --set logging.enabled=true

Hi @david.truong.devops

Usually, an HTTP 401 indicates that the license_key you have the agent configured to use in your manifest file newrelic-infrastructure-k8s-latest.yaml is tied to an account which may not have an Infrastructure entitlement or the license_key is wrong. You can double-check if you’re using the correct license_key by looking at your account settings page:

Is certainly not the incorrect license being used. I’ve verified the license key from new relic secrets and it was the exact match from the new relic account settings.

The url “https://rpm.newrelic.com/accounts/” is returning error message " Oops! You’ve found our error page."

The account I’m running is a new account (Trial). To use ‘Infrastructure’ Does it require a paid license?

Hi @david.truong.devops, I have reached out to our support team to take a further look into this and try and see what is going on. They will reach out shortly by email.