Infrastructure alert, host not reporting

Hi team,

I have a few servers with infrastructure agent installed. Suddenly, they all triggered the alert (host not reporting), however, all the servers were all up and running. Can you please take a look at the log file?

20-09-24T18:16:10-07:00" level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=951 sendErrorCount=77

time=“2020-09-24T18:16:24-07:00” level=error msg=“couldn’t post deltas” areAgentDeltas=true component=PatchSender entityKey=i-0cd0651979f6c4eef error=“Unable to submit state changes for entity [i-0cd0651979f6c4eef]: Post https://infra-api.newrelic.com/inventory/deltas: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postDeltaResults=""

time=“2020-09-24T18:16:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=952 sendErrorCount=78

time=“2020-09-24T18:17:10-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=953 sendErrorCount=79

time=“2020-09-24T18:17:26-07:00” level=warning msg=“commands poll failed” component=CommandChannelService error=“command request submission failed: Get https://infrastructure-command-api.newrelic.com/agent_commands/v1/commands: dial tcp 162.247.242.49:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

time=“2020-09-24T18:17:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=954 sendErrorCount=80

time=“2020-09-24T18:18:10-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=955 sendErrorCount=81

time=“2020-09-24T18:18:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=956 sendErrorCount=82

time=“2020-09-24T18:18:47-07:00” level=warning msg=“commands poll failed” component=CommandChannelService error=“command request submission failed: Get https://infrastructure-command-api.newrelic.com/agent_commands/v1/commands: dial tcp 162.247.242.49:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

time=“2020-09-24T18:18:54-07:00” level=error msg=“couldn’t post deltas” areAgentDeltas=true component=PatchSender entityKey=i-0cd0651979f6c4eef error=“Unable to submit state changes for entity [i-0cd0651979f6c4eef]: Post https://infra-api.newrelic.com/inventory/deltas: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postDeltaResults=""

time=“2020-09-24T18:19:10-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=957 sendErrorCount=83

time=“2020-09-24T18:19:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=958 sendErrorCount=84

time=“2020-09-24T18:20:08-07:00” level=warning msg=“commands poll failed” component=CommandChannelService error=“command request submission failed: Get https://infrastructure-command-api.newrelic.com/agent_commands/v1/commands: dial tcp 162.247.242.49:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

time=“2020-09-24T18:20:10-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=959 sendErrorCount=85

time=“2020-09-24T18:20:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=960 sendErrorCount=86

time=“2020-09-24T18:21:10-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=961 sendErrorCount=87

time=“2020-09-24T18:21:24-07:00” level=error msg=“couldn’t post deltas” areAgentDeltas=true component=PatchSender entityKey=i-0cd0651979f6c4eef error=“Unable to submit state changes for entity [i-0cd0651979f6c4eef]: Post https://infra-api.newrelic.com/inventory/deltas: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postDeltaResults=""

time=“2020-09-24T18:21:29-07:00” level=warning msg=“commands poll failed” component=CommandChannelService error=“command request submission failed: Get https://infrastructure-command-api.newrelic.com/agent_commands/v1/commands: dial tcp 162.247.242.49:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

time=“2020-09-24T18:21:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=962 sendErrorCount=88

time=“2020-09-24T18:22:10-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=963 sendErrorCount=89

time=“2020-09-24T18:22:40-07:00” level=error msg=“metric sender can’t process” component=MetricsIngestSender error=“error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)” postCount=964 sendErrorCount=90

time=“2020-09-24T18:22:50-07:00” level=warning msg=“commands poll failed” component=CommandChannelService error="command request submission failed: Get https://infrastructure-command-api.newrelic.com/agent_commands/v1/commands: dial tcp 162.247.242.49:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Thanks,
Wei

Hi @Wei.Wang-non-empl

Keep in mind that Host Not Reporting (HNR) conditions and violations aren’t really measures of whether your host is up or not, they are measures of whether your host is reporting to New Relic. So if a host’s Infrastructure agent loses connection temporarily, if it’s for a long enough period, a HNR violation may occur.

The log file you sent over looks as if connection was lost for this host. If you’d like to delve into why connection was lost, I’d loop in my colleagues who are experts in Infrastructure.

2 Likes