Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Received i/o timeout from the newrelic-infra plugin


#1

Hi,
We use newrelic-infra plugin to monitor our VM’s.
Last week, we received multiple alerts stating “Server Not Responding” and got resolved in 5 mins. Although, during this interval, servers were active and functional.
Looking at the logs, I see that newrelic-infra agent got timeout while trying to connect with newrelic endpoints.

Here are some logs.
Jun 16 10:16:34 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T10:16:34Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 16 15:25:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:25:29Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 16 15:25:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:25:59Z" level=error msg="metric sender can't process 2 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:26:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:26:29Z" level=error msg="metric sender can't process 3 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:26:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:26:59Z" level=error msg="metric sender can't process 4 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:27:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:27:29Z" level=error msg="metric sender can't process 5 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:27:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:27:59Z" level=error msg="metric sender can't process 6 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:28:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:28:29Z" level=error msg="metric sender can't process 7 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:28:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:28:59Z" level=error msg="metric sender can't process 8 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:29:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:29:29Z" level=error msg="metric sender can't process 9 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:29:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:29:59Z" level=error msg="metric sender can't process 10 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:30:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:30:29Z" level=error msg="metric sender can't process 11 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:30:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:30:59Z" level=error msg="metric sender can't process 12 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 15:31:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T15:31:29Z" level=error msg="metric sender can't process 13 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 16 16:55:47 CPID-PROD1 newrelic-infra[4737]: time="2019-06-16T16:55:47Z" level=info msg="agent has been offline for 86400000000000. Recreating delta store" component=patchSender entityID=production-dtac-apps-CPID-PROD1 Jun 18 03:41:33 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T03:41:33Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 18 03:41:53 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T03:41:53Z" level=error msg="metric sender can't process 2 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:36509->8.8.4.4:53: i/o timeout" Jun 18 03:42:13 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T03:42:13Z" level=error msg="metric sender can't process 3 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:55140->8.8.4.4:53: i/o timeout" Jun 18 03:42:33 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T03:42:33Z" level=error msg="metric sender can't process 4 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:42955->8.8.4.4:53: i/o timeout" Jun 18 03:42:53 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T03:42:53Z" level=error msg="metric sender can't process 5 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:43875->8.8.4.4:53: i/o timeout" Jun 18 05:32:39 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T05:32:39Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 18 05:32:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T05:32:59Z" level=error msg="metric sender can't process 2 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:44373->8.8.4.4:53: i/o timeout" Jun 18 05:33:19 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T05:33:19Z" level=error msg="metric sender can't process 3 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:45031->8.8.4.4:53: i/o timeout" Jun 18 05:33:39 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T05:33:39Z" level=error msg="metric sender can't process 4 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:37514->8.8.4.4:53: i/o timeout" Jun 18 12:17:54 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T12:17:54Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 18 12:18:14 CPID-PROD1 newrelic-infra[4737]: time="2019-06-18T12:18:14Z" level=error msg="metric sender can't process 2 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: dial tcp: lookup infra-api.newrelic.com on 8.8.4.4:53: read udp 192.168.100.5:58722->8.8.4.4:53: i/o timeout" Jun 19 07:54:04 CPID-PROD1 newrelic-infra[4737]: time="2019-06-19T07:54:04Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 21 04:43:48 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T04:43:48Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 21 07:17:49 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T07:17:49Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:50948->162.247.242.5:443: read: connection reset by peer" Jun 21 07:21:29 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T07:21:29Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:35824->162.247.242.5:443: read: connection reset by peer" Jun 21 07:42:09 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T07:42:09Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:35996->162.247.242.5:443: read: connection reset by peer" Jun 21 07:44:59 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T07:44:59Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:37986->162.247.242.5:443: read: connection reset by peer" Jun 21 08:04:09 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T08:04:09Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:38186->162.247.242.5:443: read: connection reset by peer" Jun 21 08:21:04 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T08:21:04Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:39514->162.247.242.5:443: read: connection reset by peer" Jun 21 08:24:08 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T08:24:08Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: read tcp 192.168.100.5:40712->162.247.242.5:443: read: connection reset by peer" Jun 21 10:39:24 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T10:39:24Z" level=error msg="metric sender can't process 1 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jun 21 10:39:39 CPID-PROD1 newrelic-infra[4737]: time="2019-06-21T10:39:39Z" level=error msg="metric sender can't process 2 times" error="Error sending events: Post https://infra-api.newrelic.com/metrics/events/bulk: net/http: TLS handshake timeout" Jun 24 09:50:38 CPID-PROD1 newrelic-infra[4737]: time="2019-06-24T09:50:38Z" level=info msg="agent has been offline for 86400000000000. Recreating delta store" component=patchSender entityID=production-dtac-apps-CPID-PROD1 (END)

Was there any incident during this interval?


#2

Hi @senthuran - The New Relic status page does not indicate any issues. Have you checked with your network team or provider?


#3

Hi @senthuran, I don’t see any official outages that would have affected this, so next step would be to take @stefan_garnham’s advice and check with your internal network team. Let us know if you find anything out :slight_smile:


#4

We’re having the very same issue, did you find out what it was? Thanks much in advance.


#5

Hi @Mariano.Barcia,

These errors are only concerning if they are repeating continuously. They just indicate network congestion to the New Relic collector

Regards,

Paul