Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon

Newrelic agent produces System Error

python
2.70
python3
uwsgi

#1

Hi,

I’m getting newrelic python agent error, frequency is pretty high.
SystemError: …/Objects/tupleobject.c:156: bad argument to internal function

During handling of the above exception, another exception occurred:

SystemError: <class 'newrelic.common.object_wrapper._NRBoundFunctionWrapper'> returned a    result with an error set 

Python 3, Newrelic 2.70.0.51, uWSGI 2.0.13.1


#2

Thank you for reporting this issue! The error appears to be generated from a cpython call made by uwsgi. We’re currently working towards finding the root cause and we will post back when we have more information.

In the meantime, there’s a couple of workarounds that you can employ.

We’ve discovered that we no longer see the issue when installing the Python agent without c-extensions. The following command will allow you to install the Python agent without the c-extensions:

NEW_RELIC_EXTENSIONS=false pip install newrelic

Please keep in mind, if you choose to do this, you need to be aware of the following caveats:

  • Although the Python agent will run, it will do so without some non-core features (i.e. The Capacity Analysis Report).
  • Part of the Python agent is written in C. Some of the agent’s code that is written in C is meant to optimize the agent. It’s possible that installing the Python agent without the C extensions may lead to a performance impact.

The other option is to change a uwsgi configuration option. This involes adding the following lines in
your uwsgi.ini file:

wsgi-env-behavior = holy

This workaround may result in a slow-growing memory leak with a large number of requests so it’s recommended that you restart your uwsgi workers periodically. In testing, we found 10,000 requests to be an appropriate limit, but feel free to increase this value if you deem it necessary. Here is one option to restart your workers (uwsgi.ini):

max-requests = 10000

If you decide to move forward with running the agent with either of these workarounds, we highly recommend that you do some performance testing to get a better idea about the performance impact.


#3

After further investigation into the issue with the uWSGI core team, the planned fix for uWSGI 2.1 (which is not yet released) to set the wsgi-env-behavior to holy by default. Until then our official recommendation is to set wsgi-env-behavior=holy manually as a workaround for this SystemError issue.

We have documented the requirement of this setting in our Python agent and uWSGI documentation.


#4

Ok, after setting wsgi-env-behavior = holy initial error is gone.
However, there are few others appeared:

   Traceback (most recent call last):
  File "/home/app/app3/lib/python3.5/site-packages/newrelic/api/web_transaction.py", line 1366, in _nr_wsgi_application_wrapper_
    result = wrapped(environ, _start_response)
  File "/home/app/app3/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 178, in __call__
    start_response(force_str(status), response_headers)
  File "/home/app/app3/lib/python3.5/site-packages/newrelic/api/web_transaction.py", line 1330, in _start_response
    additional_headers = transaction.process_response(
NameError: free variable 'transaction' referenced before assignment in enclosing scope

And:

*** Error in `/home/app/app3/bin/uwsgi': double free or corruption (fasttop): 0x0000000001a46bd0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f68f6c287e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7f68f6c30e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f68f6c3498c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xb8668)[0x7f68f726a668]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(_PyObject_GenericGetAttrWithDict+0x1e2)[0x7f68f7248ba2]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3afb)[0x7f68f736862b]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f68f736ae09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2a98)[0x7f68f7284a98]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x225a4c)[0x7f68f73d7a4c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x7e279)[0x7f68f7230279]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x83276)[0x7f68f7235276]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x1ccf)[0x7f68f73667ff]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f68f736ae09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f68f736ae09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f68f736c509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f68f736ae09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f68f736ae09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f68f736ae09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2b95)[0x7f68f7284b95]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x2d8b)[0x7f68f73678bb]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2b95)[0x7f68f7284b95]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x2d8b)[0x7f68f73678bb]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2b95)[0x7f68f7284b95]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x2d8b)[0x7f68f73678bb]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2b95)[0x7f68f7284b95]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x2d8b)[0x7f68f73678bb]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2b95)[0x7f68f7284b95]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f68f732b8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x2d8b)[0x7f68f73678bb]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f68f73fcc0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f68f73fcce3]


*** Error in `/home/app/app3/bin/uwsgi': free(): invalid pointer: 0x00007f3d58013ce0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f3d6485c7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7f3d64864e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f3d6486898c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xb8668)[0x7f3d64e9e668]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(_PyObject_GenericGetAttrWithDict+0x1e2)[0x7f3d64e7cba2]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3afb)[0x7f3d64f9c62b]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f3d65030c0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f3d65030ce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2a98)[0x7f3d64eb8a98]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f3d64f5f8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x106b04)[0x7f3d64eecb04]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(_PyObject_GenericGetAttrWithDict+0x7c)[0x7f3d64e7ca3c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3afb)[0x7f3d64f9c62b]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f3d65030c0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f3d64f9ee09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f3d65030c0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x62d9)[0x7f3d64f9ee09]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f3d65030c0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f3d65030ce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2b95)[0x7f3d64eb8b95]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f3d64f5f8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x2d8b)[0x7f3d64f9b8bb]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f3d64fa0509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x79d9)[0x7f3d64fa0509]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x24ac0c)[0x7f3d65030c0c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f3d65030ce3]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xd2a98)[0x7f3d64eb8a98]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f3d64f5f8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x225a4c)[0x7f3d6500ba4c]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x6e)[0x7f3d64f5f8ee]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f3d6502f897]
/usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x1e2ce2)[0x7f3d64fc8ce2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f3d660346ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f3d648eb82d]

Last two are from uwsgi log.
newrelic==2.82.0.62, uWSGI==2.0.14


#5

Faced the same issue. It seems the problem is gone after updating uWSGI to 2.0.15.

Python 3.5.1
newrelic==2.86.3.70
uWSGI==2.0.15


#6

Hey @kl2017 ,

You are indeed correct about the update. This was actually identified as a bug with uwsgi 2.0.14 which you can find information about here - https://github.com/unbit/uwsgi/issues/1408

This has been fixed and there’s a great breakdown on the github issue linked above.

Thanks for the update on your setup and what you were experiencing!