Not receiving any data from memcached infrastructure integration

I’ve installed the newrelic-infra agent on a test server, and have been receiving data for the host in general, but over the course of several weeks I have not received any data for the memcached integration.

I’ve installed the nri-memcached apt package, and configured it as follows:

$ sudo cat /etc/newrelic-infra/integrations.d/memcached-config.yml
integration_name: com.newrelic.memcached

# https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/memcached-monitoring-integration#config
instances:
  - name: memcached
    command: all_data
    arguments: {}

memcached is running on localhost at the default port of 11211.

When restarting the newrelic-infra service, I see this output in the systemd journal:

Apr 27 17:01:48 myhost systemd[1]: Started New Relic Infrastructure Agent.
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Starting agent process: /usr/bin/newrelic-infra"
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=infomsg="Creating Service"
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="runtime configuration" agentUser=root component="New Relic Infrastructure Agent" executablePath= maxProcs=1 pluginDir="[/etc/newrelic-infra/integrations.d /var/db/newrelic-infra/integrations.d]"
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Checking network connectivity..." component=AgentService service=newrelic-infra
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg=Initializing component=AgentService elapsedTime=372.960722ms service=newrelic-infra
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="New Relic Infrastructure Agent Running" component=AgentService elapsedTime=556.995059ms service=newrelic-infra
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Starting up agent..." component=Agent
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=metadata/attributes
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=metadata/system
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=metadata/host_aliases
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=metadata/agent_config
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=metadata/proxy_config
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=services/upstart
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=services/systemd
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=metadata/facter_facts
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=sessions/users
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=services/daemontools
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=services/supervisord
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=system/network_interfaces
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=kernel/sysctl
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=kernel/modules
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=services/pidfile
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=config/sshd
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=packages/dpkg
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin" plugin=config/selinux
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Agent plugin"plugin=metadata/cloud_security_groups
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Integration info" arguments="map[]" command=all_data commandLine="[./bin/nri-memcached]" env-vars="map[VERBOSE:0 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]" instance=localhost integration=com.newrelic.memcached interval=15 labels="map[]" os=linux prefix=config/memcached protocolVersion=2 workingDir=/var/db/newrelic-infra/newrelic-integrations
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Integration health check starting" instance=localhost integration=com.newrelic.memcached prefix=config/memcached working-dir=/var/db/newrelic-infra/newrelic-integrations
Apr 27 17:01:48 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:48-05:00" level=info msg="Integration health check finished with success" instance=localhost integration=com.newrelic.memcached prefix=config/memcached working-dir=/var/db/newrelic-infra/newrelic-integrations
Apr 27 17:01:49 myhost newrelic-infra-service[30383]: time="2020-04-27T17:01:49-05:00" level=info msg="connect got id" agent-guid=ZZZZZZ agent-id=999999999 component=IdentityConnectService

I have occasionally seen this error in the journal:

Apr 27 16:28:14 oxgoad-2019a newrelic-infra-service[22822]: time="2020-04-27T16:28:14-05:00" level=error msg="metric sender can't process" component=MetricsIngestSender error="
error sending events: Post https://infra-api.newrelic.com/infra/v2/metrics/events/bulk: read tcp 172.30.4.240:59488->162.247.242.5:443: read: connection reset by peer" postCoun
t=207 sendErrorCount=1

The error crops up occasionally, but intermittently enough that I have to assume the rest of the time the agent successfully reports to NR.

Without any other errors to report, why would the memcached integration not be reporting memcached status?

I noticed that newrelic-infra has a -debug flag, so I’ve restarted the service with the flag. This is the only difference in output I see in the logs:

Apr 28 18:00:33 oxgoad-2019a newrelic-infra-service[14913]: time="2020-04-28T18:00:33-05:00" level=info msg="Go pprof debugging enabled on http://localhost:6060" component="New Relic Infrastructure Agent"

Looking up pprof debugging, this is probably a much lower level interface than I’m comfortable with using w/o some guidance, so I’ll turn it off for now. I know with the application monitoring agent, there’s a way to have it send extra debug info to NR, but this doesn’t appear to be it.

I’ve also been able to connect to my local memcached instance and obtain stats manually:

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
stats
STAT pid 27450
STAT uptime 1663609
[...]

Hello @deyk, The log messages indicate a network issue. As the first steps in troubleshooting:

  1. Ensure you are using the latest version of the agent: Update the agent
  2. Make sure your proxy setting is correct if you are using one: Proxy variables
  3. Make sure your network is able to reach ours. Here are the endpoints: Infrastructure agents
  4. Run this command from the affected host:
curl -v https://infra-api.newrelic.com/

Let me know what your results are and we’ll go from there :slight_smile:

  1. I’ve upgraded to New Relic Infrastructure Agent version: 1.11.26 (was 1.11.21)
  2. No proxies involved
  3. NR Network is reachable. As I said, the NR dashboard for the host is displaying data. The network errors in the log appear to be only occasional blips. The only problem I’m having is receiving memcached integration data.
  4. curl works fine:
$ newrelic-infra --version                                                                                                                        [60/726]
New Relic Infrastructure Agent version: 1.11.26
[~]
[deyk@oxgoad-2019a]:~ $ curl -v https://infra-api.newrelic.com/
*   Trying 162.247.242.5...
* TCP_NODELAY set
* Connected to infra-api.newrelic.com (162.247.242.5) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):                                                                                                                                * TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):                                                                                                                        * TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=New Relic, Inc.; CN=*.newrelic.com
*  start date: Jan 19 00:00:00 2018 GMT
*  expire date: Apr 16 12:00:00 2021 GMT
*  subjectAltName: host "infra-api.newrelic.com" matched cert's "*.newrelic.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=GeoTrust RSA CA 2018
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: infra-api.newrelic.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Length: 0
< Date: Wed, 29 Apr 2020 15:04:12 GMT
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host infra-api.newrelic.com left intact

@deyk You might benefit from using New Relic Diagnostics. See if you can follow these steps and then take a look a the output messages. They will emerge in your terminal as well as in an output file:

  1. Set log_file: /var/log/newrelic-infra/newrelic-infra.log in newrelic-infra.yml (if you have not already set a location)
  2. Download the latest version of NR Diag: http://download.newrelic.com/nrdiag/nrdiag_latest.zip
  3. Unzip the file found in the linux directory
  4. From the unzipped /linux directory, move nrdiag into the Infrastructure agent root directory: /var/db/newrelic-infra
  5. Run ./nrdiag -suites infra:debug

Well for whatever reason the integration started reporting while I was away working on other things. Thanks anyway.

1 Like

That seems odd but I’m glad it’s working for you now :slight_smile: