RSyslog Events not Forwarding

Hey Everyone,

Been beating my head over this most of today and am out of anything else to try. I’ve setup a Ubuntu Server 20.04 LTS with the Infrastructure Agent that is reporting to my account just fine.

Setting up rsyslog on the system as an agentless syslog forwarder has presented no end of problems. I’m running absolutely stock following the following three documents with example configurations only changing the API Key as indicated (Note: need to put these in a comment as I’m limited to 4 links):

Below is a snipit of tail -F /var/log/syslog:

Sep  4 20:31:44 AP-LR6 68d79a28842d,U6-LR-5.60.9+12980: mcad: mcad[3300]:     wireless_agg_stats.log_sta_anomalies(): bssid=6a:d7:9a:28:84:2f radio=rai0 vap=rai2 sta=fc:a6:67:b3:f6:4e satisfaction_now=62 anomalies=tcp_latency
Sep  4 20:31:51 _gateway 928880d46055,udm-1.9.0.3413 ubios-udapi-server: ubios-udapi-server: PD address renewal (64086/64086) on br0: {
Sep  4 20:31:51 _gateway 928880d46055,udm-1.9.0.3413 ubios-udapi-server: ubios-udapi-server:  "cidr": "2600:8800:3000:ee00::1/64",
Sep  4 20:31:51 _gateway 928880d46055,udm-1.9.0.3413 ubios-udapi-server: ubios-udapi-server:  "origin": "dhcp",
Sep  4 20:31:51 _gateway 928880d46055,udm-1.9.0.3413 ubios-udapi-server: ubios-udapi-server:  "type": "dynamic",
Sep  4 20:31:51 _gateway 928880d46055,udm-1.9.0.3413 ubios-udapi-server: ubios-udapi-server:  "version": "v6"
Sep  4 20:31:51 _gateway 928880d46055,udm-1.9.0.3413 ubios-udapi-server: ubios-udapi-server: }
Sep  5 03:31:51 heimdall rsyslogd: unexpected GnuTLS error -53 - this could be caused by a broken connection. GnuTLS reports: Error in the push function.   [v8.2001.0 try /e/2078 ]
Sep  5 03:31:51 heimdall rsyslogd: omfwd: TCPSendBuf error -2078, destruct TCP Connection to newrelic.syslog.nr-data.net:6514 [v8.2001.0 try /e/2078 ]
Sep  5 03:31:51 heimdall rsyslogd: action 'action-9-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try /e/2007 ]
Sep  5 03:31:51 heimdall rsyslogd: action 'action-9-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2001.0 try /e/2359 ]

Just a simple output from my UDM-Pro to illustrate that logs are being received properly. With the following configuration in /etc/rsyslog.d/newrelic.conf:

#Define New Relic syslog format
$template NRLogFormat,"INGEST_LICENSE_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n"

# Configure TLS and log forwarding to New Relic
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net

# Configure a log stream
$InputFileName /var/log/kern.log
$InputFileTag kernel
$InputFileStateFile  kernel_log
$InputFileSeverity info
$InputRunFileMonitor

*.* @@newrelic.syslog.nr-data.net:6514;NRLogFormat

Nothing hits my NR Logs. The box can telnet to the endpoint listed in configuration:

robby@heimdall:~$ telnet newrelic.syslog.nr-data.net 6514
Trying 3.138.53.177...
Connected to newrelic.syslog.nr-data.net.
Escape character is '^]'.

I’m at a bit of a loss as to what to try next. The only “error” I can see is what’s emitted in the log above:
Sep 5 03:31:51 heimdall rsyslogd: unexpected GnuTLS error -53 - this could be caused by a broken connection. GnuTLS reports: Error in the push function. [v8.2001.0 try /e/2078 ]
Sep 5 03:31:51 heimdall rsyslogd: omfwd: TCPSendBuf error -2078, destruct TCP Connection to newrelic.syslog.nr-data.net:6514 [v8.2001.0 try /e/2078 ]
Sep 5 03:31:51 heimdall rsyslogd: action ‘action-9-builtin:omfwd’ suspended (module ‘builtin:omfwd’), retry 0. There should be messages before this one giving the reason for suspension. [v8.2001.0 try /e/2007 ]
Sep 5 03:31:51 heimdall rsyslogd: action ‘action-9-builtin:omfwd’ resumed (module ‘builtin:omfwd’) [v8.2001.0 try /e/2359 ]

However, even with that I’m not sure how to address it. Seems like a general error with the TLS connection. I’ve tried a rebuild of the system to rule out oddities with no change. Even changed my INGEST - License Key to the INGEST - License Key ID to rule out a format difference in the format.

Any tips?

Update: For anyone who runs into this issue, you MUST use an insights KEY generated from https://insights.newrelic.com/accounts/<account_id>/manage/api_keys. Assuming an ingest license will work here will generate the TLS errors mentioned above.

I’ve tried the following setup guides, practically copy and paste:

Figured out the solution, but leaving the thread open for anyone in the same situation:

You MUST use an insights KEY generated from https://insights.newrelic.com/accounts/<account_id>/manage/api_keys. Assuming an ingest license will work here will generate the TLS errors mentioned above.

2 Likes

This topic was automatically closed after 365 days. New replies are no longer allowed.