Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Newrelic airflow agent unable to send metrics

python
configuration

#1

SSL error while sending metrics to new-relic using new-relic airflow plugin.

  • We are using new relic airflow agent to send airflow metrics to new relic. (Python 3.7, airflow 1.10.4)

  • While sending data, we see following SSL error:

[2020-03-26 06:35:13,812] {{harvester.py:74}} ERROR - New Relic send_batch failed with an exception.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 472, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/newrelic_airflow_plugin/harvester.py", line 67, in _loop
    response = client.send_batch(items, common=common)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 170, in send_batch
    return self._pool.urlopen("POST", self.URL, body=payload, headers=headers)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
  • This issue is due to pyOpenSSL module being present, if the module is removed, the connection works fine and airflow-plugin is able to send data. Unfortunately, we cannot uninstall pyOpenSSL module. Is there a work-around to this problem?

#2

Hey thanks for reaching out, this is actually expected behavior. The SSL certificate received by your application when communicating with New Relic is failing to validate. urllib3 started requiring certificate verification by default for SSL connections in versions >=1.25. The preferred way to fix this issue is to install os level certificates. Check out this issue on the telemetry sdk github repo for more details and suggested workaround https://github.com/newrelic/newrelic-telemetry-sdk-python/issues/11 (newrelic-python-telemetry is what the exporter is using under the hood to send data.)


#3

Hi,
Thanks for the response.
Although, I feel the situation is slightly different.
I have OS level certs in my system, but the sdk does not pick it up.
If I change this line from:
self._pool = pool = self.POOL_CLS( host=host, port=443, retries=retries, headers=headers, strict=True )
to
self._pool = pool = self.POOL_CLS( host=host, port=443, retries=retries, headers=headers, strict=True, ca_certs='/etc/ssl/cert.pem' )
every thing starts working normally.
Also, I am hitting the issue only when pyOpenSSL lib is installed. Upon uninstalling that lib, the issue gets resloved. Alternatively, using
urllib3.contrib.pyopenssl.extract_from_urllib3()
also resolves the issue.


#4

Hi @AllDevelopers,

Just a quick follow-up on this request.

Since, you’re leveraging the New Relic Python SDK - feel free to respond directly to the commentary here on Github if you have any questions.

Also, for more context, this is a known issue, and our team are in the process of resolving this request. Again, kindly add your comments onto this open issue and you will get an update from the team directly.

Cheers!